Gather consumer lag

[Dabz]

Currently, the Metric API does not expose the consumer lag. But we could retrieve it in multiple ways, e.g. the exporter could rely on the Admin API to expose it.

[alonisser]

Note that there is lightbend/kafka-lag-exporter project that get the lag

[danksim]

@Dabz could you elaborate? is this planned work for the future? or is it already possible? would love to get Confluent Kafka broker lag metrics via this exporter.

[Dabz]

Hi @danksim , at the moment, you can not retrieve the lag via the ccloudexporter as this metric is not exposed through the Metrics API. The lag might be exposed through the Metrics API in the future, but I do not have any ETA to communicate.

In the meantime, if you need to retrieve the lag, the easiest is to rely on the lag exporter: lightbend/kafka-lag-exporter .
I did a POC a while ago to gather the lag inside ccloudexporter, it introduces a fair amount of changes (e.g. need to be bundled with librdkafka), so I put it aside for the moment.

[danksim]

@Dabz I did end up using lightbend/kafka-lag-exporter to export our hosted (paid) Confluent Kafka broker's lag metrics yesterday. Was just wondering if there was a way to do it using Dabz/ccloudexporter without introducing another dependency. That is all. Thank you!