Kafka exporter: kafka_consumergroup_lag

[robertos1232]

Kafka exporter Version 1.6
Kafka Cluster 8 brokers
topic with 8 partition.

Question:

• Why kafka exporter report in kafka_consumergroup_lag negative values?

[JasirVoriya]

The same for me.

[sky-dadan]

I have the same question.

[razorness]

I also experienced high lag spikes where summarized lag of single consumers toggle between 500 and 0 while topic receives ~20 messages per minute.

[henry-ahn0]

I have same problem. 🤔

[dramosOptiply]

How is this not given a priority yet? This causes issues in production and breaks Kubernetes HPA because it does not handle negative metric values when calculating the replica count.
This completely breaks the scaling algorithm.

[Sway23]

How is this not given a priority yet? This causes issues in production and breaks Kubernetes HPA because it does not handle negative metric values when calculating the replica count. This completely breaks the scaling algorithm.

How is this an issue with the Kafka Exporter? This is a metric your Kafka cluster is coming up with, in turn read by the Kafka Exporter.

[isaitgirl]

Same problem here
Kafka 2.6
kafka_exporter 1.8

[robertos1232]

@danielqsj Could you please take a look on this.

[dSohaliya]

I think the issue is because of we are checking for partition offset and storing it before we are calculating consumer group lag. So it might be the case that at the time of calculating consumer group lag, the consumer group offset, and partition offsets are increased. but for partition offset we are still picking old values.

Here is the PR #466
Please let me know if you find any issues here