Support dynamic attributes for metrics

[paluszkiewiczB]

Is your feature request related to a problem? Please describe.

I am using standard java.net.http.HttpClient to send a request and my application is instrumented by OpenTelemetry Java Agent with Prometheus metrics exporter.

I can see client's metrics (e.g. request duration)

I want to add attributed with name of the customer, for whom this request was executed.

Current:

http_client_duration_milliseconds_count{otel_scope_name="io.opentelemetry.java-http-client",otel_scope_version="1.30.0-alpha",http_method="POST",http_status_code="200",net_peer_name="hermes",net_peer_port="8080",net_protocol_name="http",net_protocol_version="1.1"} 5.0 1697127356980

Desired:

http_client_duration_milliseconds_count{**customer_name="first"**,otel_scope_name="io.opentelemetry.java-http-client",otel_scope_version="1.30.0-alpha",http_method="POST",http_status_code="200",net_peer_name="hermes",net_peer_port="8080",net_protocol_name="http",net_protocol_version="1.1"} 5.0 1697127356980
http_client_duration_milliseconds_count{**customer_name="second"**,otel_scope_name="io.opentelemetry.java-http-client",otel_scope_version="1.30.0-alpha",http_method="POST",http_status_code="200",net_peer_name="hermes",net_peer_port="8080",net_protocol_name="http",net_protocol_version="1.1"} 5.0 1697127356980

In my case this value is stored in org.slf4j.MDC.

Agent properties:

otel.sdk.disabled=false
otel.traces.exporter=none
otel.metrics.exporter=prometheus
otel.logs.exporter=none
otel.instrumentation.micrometer.prometheus-mode.enabled=true

Describe the solution you'd like

Extension for metrics, which would allow a 'hook' for adding dynamic attributes.

It seems to be possible for spans

Describe alternatives you've considered

For manual instrumentation I'm using custom implementation of Metrics, here is the example of LongCounter:

@ParametersAreNonnullByDefault
    public static final class MDCCounter implements LongCounter {

        private final LongCounter otel;

        public MDCCounter(LongCounter otel) {
            this.otel = otel;
        }

        @Override
        public void add(long value) {
            add(value, buildAttributes());
        }

        @Override
        public void add(long value, Attributes attrs) {
            otel.add(value, buildAttributes(attrs));
        }

        @Override
        public void add(long value, Attributes attrs, Context ctx) {
            otel.add(value, attrs, ctx);
        }

        public Attributes buildAttributes() {
            return buildAttributes(Attributes.empty());
        }

        private Attributes buildAttributes(Attributes empty) {
            String customer = MDC.get("customer_name");
            if (customer == null){
                return empty;
            }

            return Attributes.builder().put("customer_name", customer).build();
        }
    }

I've created custom implementation of ResourceProvider, which created a Resource with custom implementation of Attributes based on this example.
Unfortunately attributes are read only once, when the agent boots up (and MDC is empty at this point, so it does not work).
I'm pasting parts of implementation just to show what it looked like:

Creating resource:

@Override
  public Resource createResource(ConfigProperties config) {
    Attributes attributes = Attributes.builder().put("custom.resource", "demo").build();
    Attributes attrs = new MDCAttributes(attributes,
        MDCAttributes.fromMDC("customer_name",
            s -> new Entry<>(AttributeKey.stringKey("customer_name"), s))
    );
    return Resource.create(attrs);
  }

MDCAttributes.fromMDC

static Supplier<Map.Entry<AttributeKey<String>, String>> fromMDC(String key,
      Function<String, Map.Entry<AttributeKey<String>, String>> mapper) {
    String val = MDC.get(key);
    if (val == null) {
      return NOOP;
    }

    return () -> mapper.apply(val);
  }

MDCAttributes::forEach

 @Override
  public void forEach(BiConsumer<? super AttributeKey<?>, ? super Object> biConsumer) {
    delegate.forEach(biConsumer); // static attributes
    buildDynamic().forEach(e -> biConsumer.accept(e.getKey(), e.getValue()));
  }

Another try was to follow AutoConfigurationCustomizer example and adding the attributes through SdkMeterProvider::addResource, since Resouce::merge should add all Attributes, but this is also done once during startup.

Additional context

I know it is also possible to Create an extension with new instrumentation which injects its own advice into the same method as the original one, but I believe such use case should be supported through the SPI.

Since the examples were implemented by @iNikem, I hope you'll know whether achieving this functionality is possible in current state of the SPI :)

Many thanks for astonishing amount of work on the Agent. The amount of data it can gather 'for free' is really mind blowing 🔥

[trask]

this would help, but doesn't exist (yet): open-telemetry/oteps#207

[laurit]

Resource is immutable and constructed when the agent starts, it can't be use that way.
The way I see it your best option is to use manual instrumentation to gather the metric you need. Another options would be to customize the existing metric. For this I guess you'd need to copy the existing jdk http client instrumentation, somehow make the value from MDC available to that instrumentation (as these are in different class loader it is going to be a bit of a challenge), add the client id attribute in http client instrumentation, modify the metrics view to somehow include that attribute etc. Imo not worth the effort.

[paluszkiewiczB]

Thank you guys :)
I'll go with manual instrumentation for now and subscribe to the OTEP.