How to debug metrics not being sent to otel-collector

[mattc1170]

I've developed an application that uses the opentelemetry & opentelemetry-otlp crates to send metrics to an otel-collector. The app is successfully sending metrics when running on certain platforms (locally on my development workstation and in Google Cloud Compute Engine) but not sending metrics when running on another platform (Kubernetes cluster).

On the "bad" platform I have captured network traffic going from the application to the otel collector and see an initial TCP handshake when the connection is set up followed by a bunch of TCP keepalive transmissions, but no packets containing metrics.

Is there any sort of debugging information we can look at to figure out why metrics are not being sent on the bad system?

[ymgyt]

How about trying to use the stdout exporter?
I define the opentelemetry-stdout feature in the application and output to stdout when needed, as shown below.

use opentelemetry::{global, metrics::MeterProvider};
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_sdk::{
    metrics::{
        reader::{DefaultAggregationSelector, DefaultTemporalitySelector},
        Instrument, PeriodicReader, SdkMeterProvider, Stream, View,
    },
    runtime, Resource,
};

fn init_meter_provider(
    endpoint: impl Into<String>,
    resource: Resource,
    interval: Duration,
) -> impl MeterProvider {
    let exporter = opentelemetry_otlp::new_exporter()
        .tonic()
        .with_endpoint(endpoint)
        .build_metrics_exporter(
            Box::new(DefaultAggregationSelector::new()),
            Box::new(DefaultTemporalitySelector::new()),
        )
        .unwrap();

    let reader = PeriodicReader::builder(exporter, runtime::Tokio)
        .with_interval(interval)
        .build();

    let view = view();
    let meter_provider_builder = SdkMeterProvider::builder()
        .with_resource(resource)
        .with_reader(reader)
        .with_view(view);

    #[cfg(feature = "opentelemetry-stdout")]
    let stdout_reader = {
        let exporter = opentelemetry_stdout::MetricsExporterBuilder::default()
            .with_encoder(|writer, data| {
                serde_json::to_writer_pretty(writer, &data).unwrap();
                Ok(())
            })
            .build();
        PeriodicReader::builder(exporter, runtime::Tokio)
            .with_interval(Duration::from_secs(60))
            .build()
    };
    #[cfg(feature = "opentelemetry-stdout")]
    let meter_provider_builder = meter_provider_builder.with_reader(stdout_reader);

    let meter_provider = meter_provider_builder.build();

    global::set_meter_provider(meter_provider.clone());

    meter_provider
}

source

[lalitb]

On the "bad" platform I have captured network traffic going from the application to the otel collector and see an initial TCP handshake when the connection is set up followed by a bunch of TCP keepalive transmissions, but no packets containing metrics.

Since opentelemetry-otlp uses tonic for communication with the collector, a good first step to diagnose the issue would be to enable tracing logs to capture high-level gRPC activity (such as requests and responses). This can be done by setting up logging with tracing-subscriber and using the RUST_LOG environment variable to capture debug-level logs from tonic, hyper, and h2.

[mattc1170]

We've got a tracing-subscriber set up, though we specify targets in a config file rather than via the RUST_LOG environment variable. After adding tonic, hyper, and h2 targets all at debug level, I only get h2::codec messages in my stdout log -- nothing from tonic or hyper. However, the h2::codec messages (or lack thereof) have confirmed that we're not sending the metrics to the collector. On the good platform, we get h2::codec messages indicating that we're sending and receiving every metrics export period. We are not getting these messages on the bad platform.

[mattc1170]

I just wanted to follow up with a report on how we discovered and fixed the problem in case someone runs into a similar issue.

I wound up having to add some logging to opentelemetry-sdk where we figured out the tokio multithreaded runtime IntervalStream created in src/metrics/periodic_reader.rs was not generating the periodic Export messages that initiate metrics export. By default, the runtime will create one worker thread per CPU core. Our Kubernetes pod container was configured to use 1 CPU, so the runtime was only configured with 1 worker thread.

Something in our application was blocking that one runtime worker thread which caused the task responsible for generating the Export messages to never run. This discovery explains why there were no issues with metrics export on our other platforms -- they all had several CPU cores and therefore several runtime worker threads. The problem was fixed by configuring the runtime with multiple worker threads no matter how many CPUs were available.