Provide a way to send Liberty logs to OpenTelemetry

[donbourne]

Description

We need a way for users to be able to direct their Liberty logs to OpenTelemetry.

---

Documents

When available, add links to required feature documents. Use "N/A" to mark particular documents which are not required by the feature.

• Externally raised requests for enhancements:
  • Aha: Link to the Aha idea (also add a link to this issue in the Aha idea)
    • Feature owner adds label Aha idea
  • Open Liberty Feature Request: Link to the OL GH issue (also add a link to this issue in the Feature Request GH issue)
    • Feature owner adds label Requested feature
• UFO: https://ibm.box.com/s/ihoa8t3h18e8w2t41gdcuveo90q24mrd
• FTS: Feature Test Summary for Provide a way to send Liberty logs to OpenTelemetry #29355
• Beta Blog: BETA BLOG - Providing logs to OpenTelemetry using MicroProfile Telemetry 2.0 #29332
• GA Blog: GA BLOG - Providing log to OpenTelemetry using mpTelemetry-2.0 #29551

---

Process Overview

• Prioritization
• Design
• Implementation
• Legal and Translation
• Beta
• GA
  • Focal Point Approvals
• Other Deliverables

General Instructions

The process steps occur roughly in the order as presented. Process steps occasionally overlap.

Each process step has a number of tasks which must be completed or must be marked as not applicable ("N/A").

Unless otherwise indicated, the tasks are the responsibility of the feature owner or a delegate of the feature owner.

If you need assistance, reach out to the OpenLiberty/release-architect.

Important: Labels are used to trigger particular steps and must be added as indicated.

---

Prioritization (Complete Before Development Starts)

The OpenLiberty/chief-architect and area leads are responsible for prioritizing the features and determining which features are being actively worked on.

Prioritization

• Feature added to the "New" column of the Open Liberty project board
  • Feature owner adds label Prioritization - Requested
  • Feature added to "New" column of the Open Liberty Roadmap. (OpenLiberty/project-manager)
• Priority assigned
  • Attend the Liberty Backlog Prioritization meeting
  • Prioritization - Requested label removed (OpenLiberty/project-manager or feature owner)

---

Design (Complete Before Development Starts)

Design preliminaries determine whether a formal design, which will be provided by an Upcoming Feature Overview (UFO) document, must be created and reviewed. A formal design is required if the feature requires any of the following: UI, Serviceability, SVT, Performance testing, or non-trivial documentation/ID. Furthermore, each identified item places a blocking requirement on another team so it must be identified early in the process. The feature owner may check-off the item if they know it doesn't apply, but otherwise they should work with the focal point to determine what work, if any, will be necessary and make them aware of it.

Design Preliminaries

• UI requirements identified, or N/A. (Feature owner and UI focal point)
• Accessibility requirements identified, or N/A. (Feature owner and Accessibility focal point)
• ID requirements identified, or N/A. (Feature owner and ID focal point)
  • Refer to Documenting Open Liberty.
  • Feature owner adds label ID Required, if non-trivial documentation needs to be created by the ID team.
  • ID adds label ID Required - Trivial, if no design will be performed and only trivial ID updates are needed.
• Serviceability requirements identified, or N/A. (Feature owner and Serviceability focal point)
• SVT requirements identified, or N/A. (Feature owner and SVT focal point)
• Performance testing requirements identified, or N/A. (Feature owner and Performance focal point)

Design

• POC Design / UFO review requested.
  • Feature owner adds label Design Review Request
• POC Design / UFO review scheduled.
  • Follow the instructions in POC-Forum repo
• POC Design / UFO review completed.
• POC / UFO Review follow-ons completed.
• POC Design / UFO approval requested.
  • Feature owner adds label Design Approval Request
• Design / UFO approved. (OpenLiberty/chief-architect) or N/A
  • (OpenLiberty/chief-architect) adds label Design Approved
  • Add the public link to the UFO in Box to the Documents section.
  • The UFO must always accurately reflect the final implementation of the feature. Any changes must be first approved. Afterwards, update the UFO by creating a copy of the original approved slide(s) at the end of the deck and prepend "OLD" to the title(s). A single updated copy of the slide(s) should take the original's place, and have its title(s) prepended with "UPDATED".

No Design

• No Design requested.
  • Feature owner adds label No Design Approval Request
• No Design / No UFO approved. (OpenLiberty/chief-architect) or N/A
  • Approver adds label No Design Approved
• Feature / Capability stabilization or discontinuation or N/A
  • Feature owner adds label Product Management Approval Request and notifies OpenLiberty/product-management
  • Approver adds label Product Management Approved (OpenLiberty/product-management)
  • Note: For stabilized, superseded, and discontinued feature/capability, skip the Beta section of the template (you may delete it). Otherwise, proceed as normal.

FAT Documentation

• "Feature Test Summary" child task created
  • Use the Feature Test Summary Template
  • Add FTS issue link to the Documents section.

---

Implementation

A feature must be prioritized before any implementation work may begin to be delivered (inaccessible/no-ship). However, a design focused approach should still be applied to features, and developers should think about the feature design prior to writing and delivering any code.
Besides being prioritized, a feature must also be socialized (or No Design Approved) before any beta code may be delivered. All new Liberty content must be inaccessible in our GA releases until it is Feature Complete by either marking it kind=noship or beta fencing it.
Code may not GA until this feature has obtained the Design Approved or No Design Approved label, along with all other tasks outlined in the GA section.

Feature Development Begins

• Add the In Progress label

Legal and Translation

In order to avoid last minute blockers and significant disruptions to the feature, the legal items need to be done as early in the feature process as possible, either in design or as early into the development as possible. Similarly, translation is to be done concurrently with development. All items below MUST be completed before beta & GA is requested.

Innovation (Complete 1 week before Beta & GA Feature Complete Date)

• Consider whether any aspects of the feature may be patentable. If any identified, disclosures have been submitted.

Legal (Complete before Beta & GA Feature Complete Date)

• Changed or new open source libraries are cleared and approved, or N/A. (Legal Release Services/Cass Tucker/Release PM).

Translation (Complete by Beta & GA Feature Complete Date)

• PII (Program Integrated Information) updates are merged (i.e. all English strings due for translation have been delivered), or N/A.

---

Beta

In order to facilitate early feedback from users, all new features and functionality should first be released as part of a beta release.

Beta Code

• Beta fence the functionality
  • E.g. kind=beta, ibm:beta, ProductInfo.getBetaEdition()
• Beta development complete and feature ready for inclusion in a beta release
  • Add label target:beta and the appropriate target:YY00X-beta (where YY00X is the targeted beta version).
• Feature delivered into beta
  • (OpenLiberty/release-manager) adds label release:YY00X-beta (where YY00X is the first beta version that included the functionality).

Beta Blog (Complete by beta eGA)

• Beta blog issue created and populated using the Open Liberty BETA blog post template.
  • Add a link to the beta blog issue in the Documents section.
  • Note: This is for inclusion into the overall beta release blog post. If, in addition, you'd also like to create a dedicated blog post about your feature, then follow the "Standalone Feature Blog Post" instructions under the Other Deliverables section.

---

GA

A feature is ready to GA after it is Feature Complete and has obtained all necessary Focal Point Approvals.

Feature Complete

• Feature implementation and tests completed.
  • All PRs are merged.
  • All related/child issues are closed.
  • All stop ship issues are completed.
• Legal: all necessary approvals granted.
• Innovation: IP identified and any applicable disclosures submitted
• Translation: Feature may only proceed to GA if it has either Translation - Complete or Translation - Missing label
  • If all translation has been delivered to release branch, feature owner adds label Translation - Complete.
  • If missing translation does not cause a break in functionality, nor a security or production outage risk, feature owner adds label Translation - Missing.
    • Once all missing translations are delivered, the Translation - Missing label is replaced with Translation - Complete.
  • If missing translation could cause a break in functionality or a security or production outage risk, feature owner adds the Translation - Blocked label.
    • Features with Translation - Blocked may NOT proceed to GA until the label has been replaced with either Translation - Missing or Translation - Complete.
  • For further guidance, contact Globalization focal point or the Release Architect.
• GA development complete and feature ready for inclusion in a GA release
  • Add label target:ga and the appropriate target:YY00X (where YY00X is the targeted GA version).
  • Inclusion in a release requires the completion of all Focal Point Approvals.

Focal Point Approvals (Complete by Feature Complete Date)

These occur only after GA of this feature is requested (by adding a target:ga label). GA of this feature may not occur until all approvals are obtained.

All Features

• APIs/Externals - Externals have been reviewed or N/A. (OpenLiberty/externals-approvers)
  • Approver adds label focalApproved:externals
• Demo - Demo is scheduled for an upcoming EOI or N/A. (OpenLiberty/demo-approvers)
  • Add comment @OpenLiberty/demo-approvers Demo scheduled for EOI [Iteration Number] to this issue.
  • Approver adds label focalApproved:demo.
• FAT - All Tests complete and running successfully in SOE or N/A. (OpenLiberty/fat-approvers)
  • Approver adds label focalApproved:fat.

Design Approved Features

• ID - Documentation is complete or N/A. (OpenLiberty/id-approvers)
  • Approver adds label focalApproved:id.

  • NOTE: If only trivial documentation changes are required, you may reach out to the ID Feature Focal to request a ID Required - Trivial label. Unlike features with regular ID requirement, those with ID Required - Trivial label do not have a hard requirement for a Design/UFO.

• InstantOn - InstantOn capable or N/A. (OpenLiberty/instantOn-approvers)
  • Approver adds label focalApproved:instantOn.
• Performance - Performance testing is complete or N/A. (OpenLiberty/performance-approvers)
  • Approver adds label focalApproved:performance.
• Serviceability - Serviceability has been addressed or N/A. (OpenLiberty/serviceability-approvers)
  • Approver adds label focalApproved:sve.
• STE - Skills Transfer Education chart deck is complete or N/A. (OpenLiberty/ste-approvers)
  • Approver adds label focalApproved:ste.
• SVT - System Verification Test is complete or N/A. (OpenLiberty/svt-approvers)
  • Approver adds label focalApproved:svt.

Remove Beta Fencing (Complete by Feature Complete Date)

• Beta guards are removed, or N/A
  • Only after all necessary Focal Point Approvals have been granted.

GA Blog (Complete by Friday after GM)

• GA Blog issue created and populated using the Open Liberty GA release blog post template.
  • Add a link to the GA Blog issue in the Documents section.
  • Note: This is for inclusion into the overall release blog post. If, in addition, you'd also like to create a dedicated blog post about your feature, then follow the "Standalone Feature Blog Post" instructions under the Other Deliverables section.

Post GM (Complete before GA)

• After confirming this feature has been included in the GM driver, feature owner closes this issue.

Post GA

• Remove the target:ga and target:YY00X labels, and add the appropriate release:YY00X. (OpenLiberty/release-manager)

---

Other Deliverables

• Standalone Feature Blog Post - A blog post specifically about your feature or N/A. (Feature owner and OpenLiberty/release-architect)
  • This should be strongly considered for larger or more prominent features.
  • Follow instructions in the blogs repo.
• OL Guides - OL Guides assessment is complete or N/A. (OpenLiberty/guide-assessment)
• Dev Experience - Developer Experience & Tools work is complete or N/A. (OpenLiberty/dev-experience-assessment)

---

[pgunapal]

Design [DRAFT] :

*** Subject to change ***

High Level User Story:
As an Operations engineer, I want to be able to export logs from Open Liberty to an Open Telemetry Exporter.

OpenTelemetry defines a Logs Bridge API for emitting LogRecords. OpenTelemetry provides a Logs Bridge API and SDK, which can be used together with existing logging libraries to automatically inject the trace context in the emitted logs, and provide an easy way to send the logs via OTLP. Instead of modifying each logging statement, log appenders use the API to bridge logs from existing logging libraries to the OpenTelemetry data model, where the SDK controls how the logs are processed and exported. The typical log SDK configuration installs a log record processor and exporter.

The LogRecordProcessor from the Logs SDK allows us to process and decorate the LogRecord fields to map to OTel Log Data Model.

• SimpleLogRecordProcessor: This is an implementation of LogRecordProcessor which passes finished logs and passes the export-friendly ReadableLogRecord representation to the configured LogRecordExporter, as soon as they are finished.

• BatchLogRecordProcessor: This is an implementation of the LogRecordProcessor which create batches of LogRecords and passes the export-friendly ReadableLogRecord representations to the configured LogRecordExporter.

BatchLogRecordProcessor and SimpleLogRecordProcessor are paired with LogRecordExporter, which is responsible for sending telemetry data to a particular backend.

Feature Design:

In Open Liberty, Open Telemetry is initialized using the SDK autoconfiguration extension, instead of manually creating the OpenTelemetry instance by using the SDK builders directly in the code. This approach allows you to autoconfigure the OpenTelemetry SDK based on a standard set of supported environment variables and system properties. Hence, the logging providers can be configured using environment variables.
Ref: https://opentelemetry.io/docs/languages/java/instrumentation/#autoconfiguration

Since, the mpTelemetry-2.0 OpenTelemetry instance is dependent on application thread context, it will be difficult to get the instance, when we are not apart of the application thread context, such as during server start up. Hence, we need to explicitly create a new server-level Open Telemetry instance, which would have its own server-specific OTel configuration. This will also work for the multi-app scenarios as well.

• Enable the mpTelemetry-2.0 feature and configure the Environment variables to enable the server-level OTel SDK and configure the OTel Log exporter and LogRecord Processors, as follows:

mp.server.otel.service.name (can be similar as app-level config)
mp.server.otel.sdk.disabled=false
mp.server.otel.logs.exporter=otlp
mp.server.otel.exporter.otlp.endpoint=http://localhost:4317/

There will be server-level resource attributes configured for the server-level OpenTelemetry instance, as well.
(https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection)

By default, the SimpleLogRecordProcessor will be enabled, where the records will be send immediately. However, if you want to send the records in batches, you can also configure the following logging specific Batch LogRecord Processor Environment variables to configure how often and how to export the logs over, and as well as log record limits for attributes.
(https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#batch-logrecord-processor)

OTEL_BLRP_SCHEDULE_DELAY : Delay interval (in milliseconds) between two consecutive exports (Default = 1000)
OTEL_BLRP_EXPORT_TIMEOUT : Maximum allowed time (in milliseconds) to export data (Default = 30000)
OTEL_BLRP_MAX_QUEUE_SIZE : Maximum queue size (Default = 2048)
OTEL_BLRP_MAX_EXPORT_BATCH_SIZE : Maximum batch size (Default = 512)

OTEL_LOGRECORD_ATTRIBUTE_VALUE_LENGTH_LIMIT : Maximum allowed attribute value size (Default = no limit)
OTEL_LOGRECORD_ATTRIBUTE_COUNT_LIMIT : Maximum allowed log record attribute count (Default = 128)

• Use the mpTelemetry-2.0 feature in server.xml, with server configuration attribute to configure the different log sources (message, trace, accessLog, ffdc, garbageCollection, and audit) we want to send to OTel exporter. Default value for the logSources attribute would be message.

<feature>mpTelemetry-2.0</feature>
…
<mpTelemetry logSources=“message, trace, accessLog, ffdc, audit”/>

• The feature design would be similar to the Logstash Collector 1.0 feature, where we will be subscribing a “OTel Handler” to the collector manager, where we would get access to each event and format the event accordingly (JSONify), to map to the OTel Log Data Model and then emit it to the “Target”, which would be the configured exporter for OTel logs.

• The corresponding SpanID and TraceID for trace context and application will be retrieved from the LogRecordContext (ext_traceID and ext_spanID).

Mapping Open Liberty Log Record to Open Telemetry Logs Data Model
(https://opentelemetry.io/docs/specs/otel/logs/data-model/#log-and-event-record-definition)
Note: When formatting the event, JSONify the event, so the event is structured properly.

Open Liberty Log Record | Open Telemetry Logs Data Model
=============================================
ibm_datetime = Timestamp
ext_traceId = TraceId
ext_spanId = SpanId
loglevel = SeverityText
(Refer to table below) = SeverityNumber 
message  = Body *** Should be the entire JSON payload instead?
host = Resource[“host.name”]
service.name = Resource[“service.name”]
io.openliberty.microprofile.telemetry = InstrumentationScope
Map the rest of the fields as Key:Value pairs = Attributes[“Key”]
Throwable example snippet:
throwable.getClass().getName() = Attributes[SemanticAttributes.EXCEPTION_TYPE]
throwable.getMessage = Attributes[SemanticAttributes.EXCEPTION_MESSAGE]
throwable.printStackTrace() = Attributes[SemanticAttributes.EXCEPTION_STACKTRACE]

Log Level Mapping to Open Telemetry Severity Text
(https://opentelemetry.io/docs/specs/otel/logs/data-model/#severity-fields)
(https://opentelemetry.io/docs/specs/otel/logs/data-model-appendix/#appendix-b-severitynumber-example-mappings)

Open Liberty Log Level | Open Telemetry Logs Severity Text / Number
====================================================
FATAL = FATAL / 21
SEVERE = ERROR / 17
WARNING = WARN / 13
AUDIT = INFO2 / 10
INFO = INFO / 9
CONFIG = DEBUG4 / 8
DETAIL = DEBUG3 / 7
FINE = DEBUG2 / 6
FINER = DEBUG / 5
FINEST = TRACE / 1

[pgunapal]

High-level Implementation Design Details [DRAFT] :

*** Subject to change ***

• Update mpTelemetry-2.0 feature files (public and private, for different EE versions) to include project io.openliberty.microprofile.telemetry.2.0.logging.internal
• Add the required OpenTelemetry Logs SDK and API libraries to the mpTelemetry-2.0 feature file:

io.opentelemetry.sdk.logs.export;type="third-party",\
io.opentelemetry.sdk.logs.data;type="third-party",\
io.opentelemetry.api.logs;type="third-party",\

• Create a new class io.openliberty.microprofile.telemetry20.OpenTelemetryHandler, in the new project io.openliberty.microprofile.telemetry.2.0.logging.internal, that extends com.ibm.ws.collector.Collector, similar to how its done in LogstashCollector.
• The contents of OpenTelemetryHandler class and the BND files should be similar to the LogstashCollector project, however, omit the redundant components, such as SSL, ExecutorService, which will not be needed for this feature.
• Add the corresponding mpTelemetry-2.0 packages to the BND file.

Import-Package:
io.openliberty.microprofile.telemetry.internal.common

-buildpath:
	io.openliberty.microprofile.telemetry.internal.common;version=latest,\
	io.openliberty.io.opentelemetry.2.0;version=latest

• Ensure the correct metatype is defined for the server configuration of mpTelemetry-2.0 (e.g. logSources)
• Update the OpenTelemetryVersionedConfigurationImpl class file to remove the following lines, since we should be enabling Logs, as part of this feature.

telemetryProperties.put(OpenTelemetryConstants.CONFIG_LOGS_EXPORTER_PROPERTY, "none");
telemetryProperties.put(OpenTelemetryConstants.ENV_LOGS_EXPORTER_PROPERTY, "none");

• In the OpenTelemetryHandler.activate() method, retrieve and set the server-level OpenTelemetryInfo object. It will be using the OpenTelemetryAccessor interface from the io.openliberty.microprofile.telemetry.internal.common project (TBD - details to follow from MP Telemetry team)
• Get the configured OpenTelemetry LogProvider by calling the OpenTelemetryInfo.getOpenTelemetry().getLogsBridge(). (https://javadoc.io/doc/io.opentelemetry/opentelemetry-api/latest/io/opentelemetry/api/OpenTelemetry.html)
• In the OpenTelemetryHandler.formatEvents() method, by using the previously retrieved LogProvider, get the LogBuilder (https://javadoc.io/doc/io.opentelemetry/opentelemetry-api/latest/io/opentelemetry/api/logs/LoggerProvider.html), with the configured instrumentation name, and use the builder to map the Open Liberty event records to the appropriate OpenTelemetry Log Data model.
• Once, the builder is mapped with the corresponding fields, call builder.emit() to export the logs to the exporter.

Below is a generic high-level code snippet on how to retrieve the OpenTelemetry LogProvider/builder, and to map generic JUL Log Record fields to Open Telemetry Log Data Model, and then to export it to the configured exporter.

Snippet is from the Open Telemetry JUL Java agent instrumentation :

...
String instrumentationName = logger.getName();
    if (instrumentationName == null || instrumentationName.isEmpty()) {
      instrumentationName = "ROOT";
    }
    LogRecordBuilder builder =
        GlobalOpenTelemetry.get()
            .getLogsBridge()
            .loggerBuilder(instrumentationName)
            .build()
            .logRecordBuilder();
    mapLogRecord(builder, logRecord);
    builder.emit();

private static void mapLogRecord(LogRecordBuilder builder, LogRecord logRecord) {
    // message
    String message = FORMATTER.formatMessage(logRecord);
    if (message != null) {
      builder.setBody(message);
    }

    // time
    long timestamp = logRecord.getMillis();
    builder.setTimestamp(timestamp, TimeUnit.MILLISECONDS);

    // level
    Level level = logRecord.getLevel();
    if (level != null) {
      builder.setSeverity(levelToSeverity(level));
      builder.setSeverityText(logRecord.getLevel().getName());
    }

    AttributesBuilder attributes = Attributes.builder();

    // throwable
    Throwable throwable = logRecord.getThrown();
    if (throwable != null) {
      attributes.put(SemanticAttributes.EXCEPTION_TYPE, throwable.getClass().getName());
      attributes.put(SemanticAttributes.EXCEPTION_MESSAGE, throwable.getMessage());
      StringWriter writer = new StringWriter();
      throwable.printStackTrace(new PrintWriter(writer));
      attributes.put(SemanticAttributes.EXCEPTION_STACKTRACE, writer.toString());
    }

    if (captureExperimentalAttributes) {
      Thread currentThread = Thread.currentThread();
      attributes.put(SemanticAttributes.THREAD_NAME, currentThread.getName());
      attributes.put(SemanticAttributes.THREAD_ID, currentThread.getId());
    }

    builder.setAllAttributes(attributes.build());

    // span context
    builder.setContext(Context.current());
  }
...

Note: OpenTelemetry have implemented Log Appenders Log4J and Logback.

[benjamin-confino]

Hello.

In the OpenTelemetryHandler.activate() method, retrieve and set the server-level OpenTelemetryInfo object. It will be using the OpenTelemetryAccessor interface from the io.openliberty.microprofile.telemetry.internal.common project (TBD - details to follow from MP Telemetry team)

This should work fine but there is one hidden gotcha to be aware off. In OpenTelemetryInfoFactory we have a check when an OpenTelemetryInfo is created if (j2EEName.startsWith("io.openliberty") || j2EEName.startsWith("com.ibm.ws")) { you will need to make sure this if statement returns false when OpenTelemetryHandler calls us. I suspect without modification it will be true.

[pgunapal]

Thanks @benjamin-confino ! Good point, will make sure that doesn't break for us. Right, getOpenTelemetryInfo() will be called by internal code, so it would be true.

[chirp1]

Slack with Prashanth, David, Ram, me. Prashanth provided the necessary info in the following doc issue: OpenLiberty/docs#7459. I approved the feature.

[tngiang73]

@pgunapal: WASWIN is good with the STE slides. STE approved.

[donbourne]

OL:

Serviceability Approval Comment - Please answer the following questions for serviceability approval:

1. UFO -- does the UFO identify the most likely problems customers will see and identify how the feature will enable them to diagnose and solve those problems without resorting to raising a PMR? Have these issues been addressed in the implementation?

2. Test and Demo -- As part of the serviceability process we're asking feature teams to test and analyze common problem paths for serviceability and demo those problem paths to someone not involved in the development of the feature (eg. IBM Support, test team, or another development team).
   a) What problem paths were tested and demonstrated?
   b) Who did you demo to?
   c) Do the people you demo'd to agree that the serviceability of the demonstrated problem scenarios is sufficient to avoid PMRs for any problems customers are likely to encounter, or that IBM Support should be able to quickly address those problems without need to engage SMEs?

3. SVT -- SVT team is often the first team to try new features and often encounters problems setting up and using them. Note that we're not expecting SVT to do full serviceability testing -- just to sign-off on the serviceability of the problem paths they encountered.
   a) Who conducted SVT tests for this feature?
   b) Do they agree that the serviceability of the problems they encountered is sufficient to avoid PMRs, or that IBM Support should be able to quickly address those problems without need to engage SMEs?

4. Which IBM Support / SME queues will handle PMRs for this feature? Ensure they are present in the contact reference file and in the queue contact summary, and that the respective IBM Support/SME teams know they are supporting it. Ask Don Bourne if you need links or more info.

5. Does this feature add any new metrics or emit any new JSON events? If yes, have you updated the JMX metrics reference list / Metrics reference list / JSON log events reference list in the Open Liberty docs?

[donbourne]

@clarkek123 will be handling the serviceability approval for this epic.

[dave-waddling]

Thanks for completing the FTS. The results of the mini-SOE look good so adding the FAT Focal approval.

[pgunapal]

@clarkek123 I have filled out the template below, can you please review the Serviceability approval for this feature?

Serviceability Approval Comment - Please answer the following questions for serviceability approval:

UFO -- does the UFO identify the most likely problems customers will see and identify how the feature will enable them to diagnose and solve those problems without resorting to raising a PMR? Have these issues been addressed in the implementation?

A: Yes. For the logging component of the mpTelemetry-2.0, the logs should be exported automatically if the mpTelemetry-2.0 feature is enabled in the server.xml, along with the otel.sdk.disabled=true is configured either in Liberty or the application, and the otel.logs.exporter is set to a valid exporter. Below are some of the common user error scenarios for logs not being exported that we tested in FATs, as well as manually.

• otel.logs.exporter is set to none. – A warning message is shown stating that the exporter for logs is disabled.
• otel.sdk.disabled is not set to false. – A warning message is shown that the OpenTelemetry SDK instance is not enabled.
• The source attribute is empty in the server.xml. (), a warning message will be shown.
• If there are invalid log event sources configured, a warning message will be logged: CWMOT5005W: The MicroProfile Telemetry logging feature does not recognize the [ {0} ] log source.

Test and Demo -- As part of the serviceability process we're asking feature teams to test and analyze common problem paths for serviceability and demo those problem paths to someone not involved in the development of the feature (eg. IBM Support, test team, or another development team).
a) What problem paths were tested and demonstrated?
b) Who did you demo to?
c) Do the people you demo'd to agree that the serviceability of the demonstrated problem scenarios is sufficient to avoid PMRs for any problems customers are likely to encounter, or that IBM Support should be able to quickly address those problems without need to engage SMEs?

A:
a) The problems paths mentioned above were also demoed to analyze the problem paths.
b) Local, EOI, SVT, Performance teams.
c) Yes

SVT -- SVT team is often the first team to try new features and often encounters problems setting up and using them. Note that we're not expecting SVT to do full serviceability testing -- just to sign-off on the serviceability of the problem paths they encountered.
a) Who conducted SVT tests for this feature?
b) Do they agree that the serviceability of the problems they encountered is sufficient to avoid PMRs, or that IBM Support should be able to quickly address those problems without need to engage SMEs?

A:
a) Daniel Guinan
b) Yes

Which IBM Support / SME queues will handle PMRs for this feature? Ensure they are present in the contact reference file and in the queue contact summary, and that the respective IBM Support/SME teams know they are supporting it. Ask Don Bourne if you need links or more info.

A:
WAS L3: Logging

Does this feature add any new metrics or emit any new JSON events? If yes, have you updated the JMX metrics reference list / Metrics reference list / JSON log events reference list in the Open Liberty docs?

A:
No

[clarkek123]

Based on the information provided above for Serviceability showing common error paths testing with demo with approval from Local, SVT and Perfomance teams and SVT signoff on the paths included for serviceability, I have added the Serviceability approval for this feature.

[pgunapal]

Closing, as this feature is complete, and delivered for 24.0.0.9.