Prometheus instrumentation library for confluent-kafka-python applications.
Plain and simple, it implements a metrics mapper which translate confluent-kafka's native metrics to prometheus format.
This python package is released at PyPi: https://pypi.org/project/prometheus-kafka-metrics/.
Release is automated using git-webhook which triggers build and deployment at CircleCI pipeline(https://app.circleci.com/pipelines/github/shakti-garg/prometheus_kafka_metrics)
- Instantiate the relevant metrics manager and pass its send function as parameter to the kafka-client's callback
stats_cb.
For confluent-kafka producer:
from confluent_kafka import Producer
from prometheus_kafka_producer.metrics_manager import ProducerMetricsManager
metric_manager = ProducerMetricsManager()
conf = {'bootstrap.servers': brokers,
'stats_cb': metric_manager.send
}
Producer(conf)
Similary, for confluent-kafka consumer:
from confluent_kafka import Consumer
from prometheus_kafka_consumer.metrics_manager import ConsumerMetricsManager
metric_manager = ConsumerMetricsManager()
conf = {'bootstrap.servers': brokers,
'group.id': consumer_group_id,
'stats_cb': metric_manager.send
}
Consumer(conf)
- Publish the generated prometheus metrics at a http port. In case you are not using any relevant framework, it can be done naively like below:
from prometheus_client import start_http_server
if __name__ == '__main__':
start_http_server(8000)