Prometheus not working with statsd metrics

[bigman73]

I followed the getting started demo.
The metrics show up in Prometheus.

I then changed the receiver to statsd and exposed port 8125 udp on docker compose. When I send custom metrics I see them logged in the console but they don't show up in Prometheus, even after waiting.

otel-collector-config.yaml

receivers:
  statsd:
    endpoint: "0.0.0.0:8125" # default
    aggregation_interval: 10s

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    const_labels:
      label1: value1

  logging:
    verbosity: detailed

  file:
    path: ./metrics.log

processors:
  batch:

extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    metrics:
      receivers: [statsd]
      processors: [batch]
      exporters: [file, logging, prometheus]

docker-compose.yml:

version: "2"
services:
  # Collector
  otel-collector:
    image: ${OTELCOL_IMG}
    restart: always
    command: ["--config=/etc/otel-collector-config.yaml", "${OTELCOL_ARGS}"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
      - ./metrics.log:/metrics.log
    ports:
      - "1888:1888"   # pprof extension
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "13133:13133" # health_check extension
      - "4317:4317"   # OTLP gRPC receiver
      - "55679:55679" # zpages extension
      
      - "8125:8125/udp"   # StatsD receiver *****

  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    restart: always
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

Test script:

echo "testmetric:42|c|#myKey:myVal" | nc -w 1 -u localhost 8125

Console output - this is how I know the statsd receiver works:
metrics-otel-collector-1 | 2022-12-21T17:11:58.646Z info ResourceMetrics #0
metrics-otel-collector-1 | Resource SchemaURL:
metrics-otel-collector-1 | ScopeMetrics #0
metrics-otel-collector-1 | ScopeMetrics SchemaURL:
metrics-otel-collector-1 | InstrumentationScope
metrics-otel-collector-1 | Metric #0
metrics-otel-collector-1 | Descriptor:
metrics-otel-collector-1 | -> Name: testmetric
metrics-otel-collector-1 | -> Description:
metrics-otel-collector-1 | -> Unit:
metrics-otel-collector-1 | -> DataType: Sum
metrics-otel-collector-1 | -> IsMonotonic: false
metrics-otel-collector-1 | -> AggregationTemporality: Delta
metrics-otel-collector-1 | NumberDataPoints #0
metrics-otel-collector-1 | Data point attributes:
metrics-otel-collector-1 | -> myKey: Str(myVal)
metrics-otel-collector-1 | StartTimestamp: 2022-12-21 17:10:58.533688167 +0000 UTC
metrics-otel-collector-1 | Timestamp: 2022-12-21 17:11:09.390385089 +0000 UTC
metrics-otel-collector-1 | Value: 42
metrics-otel-collector-1 | {"kind": "exporter", "data_type": "metrics", "name": "logging"}

The metrics.log file also gets populated, as expected.

Any ideas?

[platoyuval]

Prometheus does show the OTEL 'stock' metrics, but not the custom metric I was reporting.

Is there some configuration I need to add?

[jpkrohling]

Looks like your scrape config is scraping only the port 8888, which contains the metrics related to the otelcol process. Are you also scraping the port 8889, which is your prometheus exporter port?

[platoyuval]

Thanks, that could have been it
I moved away from this approach since then. But we'll check.

[swar8080]

The statsd receiver emits metrics with delta temporarily, which the prometheus exporter drops (since it only support cumulative temporality):

https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality

This config will use cumulative temporarily, but I didn't explore how accurate the data is in prometheus

statsd:
    is_monotonic_counter: true