Awesome Open-Source Contribs for Apache Kafka

A curated list of awesome open-source frameworks, libraries, tools and examples for the Apache Kafka project.

Contributing to this project is easy!Just send a pull request and make the list growth!

Table of Contents

• Change Data Capture
• Clients
• CLI / Tools
• Configuration
• Examples
• Infrastructure
• Kafka Connect
• Kafka Streams
• Performance
• Query/SQL
• Replication
• REST API
• Schema / Data
• Security / ACL
• Site Reliability Tools
• Test
• UI/Cluster Management
• Other Awesome Lists about Apache Kafka & Data Engineering
• License

Change Data Capture

• Debezium: Debezium is an open source project that provides a low latency data streaming platform for change data capture (CDC). Apache License 2.0

• Maxwell: Maxwell’s daemon, a mysql-to-json kafka producer. Apache License 2.0

Clients

• kafkajs: A modern Apache Kafka client for node.js MIT License

• kafka-clients-kotlin: This projects gives Kotlin bindings and several extensions for Apache Kafka Clients.

• Kafka Rust: Rust client for Apache Kafka. MIT License

• librdkafka: The Apache Kafka C/C++ library. 2-clause BSD license

• Sarama: Sarama is a Go library for Apache Kafka 0.8, and up. MIT License

• franz-go Franz-go is a feature complete client written in native Go. BSD 3-Clause License

CLI / Tools

• [Firehose](https://github.com/odpf/firehose) - Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.

• Kafkacat: Generic command line non-JVM Apache Kafka producer and consumer. librdkafka - Apache Kafka C driver library

• kafkacli: CLI and Go Clients to manage Kafka components (Kafka Connect & SchemaRegistry). Apache License 2.0

• Kafka Shell: ⚡A supercharged, interactive Kafka shell built on top of the existing Kafka CLI tools. Apache License 2.0

• kukulcan: A REPL for Apache Kafka. Apache License 2.0

• Zoe: The Kafka CLI for humans. MIT License

Configuration

• kafka-gitops: Manage Apache Kafka topics and generate ACLs through a desired state file. Apache License 2.0

• Jikkou: Tool to ease and automate Apache Kafka cluster configuration management. Apache License 2.0

• kattlo: Kattlo gives a way to manage topics, ACLs, users, schemas, ksqlDB, employing a evolutionary configuration. Easy to understand, easy to write and easy to apply. Apache License 2.0

• julie: An operational manager for Apache Kafka (Automation, GitOps, SelfService) MIT License

Examples

• Confluent demo-scene: 👾 Scripts and samples to support Confluent Demos and Talks.

• Confluent Kafka Streams Examples: Demo applications and code examples for Apache Kafka’s Streams API. Apache License 2.0

• Confluent Examples: Apache Kafka and Confluent examples and demos. Apache License 2.0

• kafka-examples: Snippets and small examples demonstrating kafka features and configs. Apache License 2.0

Infrastructure

• Confluent Docker Images: Docker images for Confluent Platform. [Apache License 2.0](Apache License 2.0)

• Confluent Platform Ansible: Ansible playbooks for the Confluent Platform. Apache License 2.0

• Strimzi: Strimzi provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations. Apache License 2.0

Kafka Connect

• kafka-connect-client: Java client library for Apache Kafka Connect REST Interface. Apache License 2.0

• kcctl: A command line client for Kafka Connect Apache License 2.0

Kafka Streams

• Azkarra Streams: Azkarra Streams is a lightweight Java framework which makes easy to develop and operate Kafka Streams applications. Apache License 2.0

• Faust: A library for building streaming applications in Python. BSD License

• GoKa: Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go. BSD 3-Clause "New" or "Revised" License

• Kafka Streams CEP: Complex Event Processing on top of Kafka Streams (Apache License 2.0

• Kafka Streams .NET: .NET Stream Processing Library for Apache Kafka

• Kafka Streams Viz: A tool helps visualize stream topologies by generating nice looking diagrams from a kafka stream topology descriptions.

• [Dagger](https://github.com/odpf/dagger) - Dagger is an easy-to-use, SQL framework to process built on top of Apache Flink for stateful pro

Performance

• kloadgen: KLoadGen is kafka load generator plugin for jmeter designed to work with AVRO schema Registries. MPL 2.0

Query/SQL

• ksqlDB: The event streaming database purpose-built for stream processing applications. Confluent Community License Agreement Version 1.0

• ksql-jdbc-driver: JDBC driver for Apache Kafka. Apache License 2.0

Replication

• Mirror Maker 2: MM2 leverages the Connect framework to replicate topics between Kafka clusters. Apache License 2.0

• Uber uReplicator: Improvement of Apache Kafka Mirrormaker. Apache License 2.0

REST API

• Kafka REST Proxy: Confluent REST Proxy for Kafka. Confluent Community License Agreement Version 1.0

• Karapace: Karapace (Aiven) is the open source REST proxy you can run where you need it; drop-in replacement for Confluent REST Proxy. Apache License 2.0

• Kafka Pixy: gRPC/REST proxy for Kafka. Apache License 2.0

• Raccoon - Raccoon is a high-throughput, low-latency service to collect events in real-time from your web, mobile apps, and services using multiple network protocols and publish to Kafka.

• Zilla - An API gateway built for event-driven architectures and streaming that supports standard protocols such as HTTP, SSE, gRPC, MQTT, and the native Kafka protocol.

Schema / Data

• [Stencil](https://github.com/odpf/stencil) - Stencil is a Protobuf schema registry for Kafka that enables you to create, maintain and consume protobuf messages and APIs dynamically, efficiently, and reliably.

• Javro: Javro is an interface allowing you to write Avro schemas with ease, to validate it, and to preview it into JSON format with samples. Gpl 3.0

• Confluent Schema Registry: Confluent Schema Registry for Kafka. Confluent Community License Agreement Version 1.0

• Karapace: Karapace (Aiven) is the open source schema registry you can run on your own platforms. Alternative (compatible replacement) for Confluent Schema Registry, supports Avro and JSON Schema. [Apache License 2.0

Security / ACL

• kafka-security-manager: Manage your Kafka ACL at scale. MIT License

• kafka-encryption: About Kafka End to End Encryption. Apache License 2.0

Site Reliability Tools

• Burrow(Linkedin): Kafka Consumer Lag Checking. Apache License 2.0

• Chaperone( Uber): As Kafka audit system, Chaperone monitors the completeness and latency of data stream.The audit metrics are persisted in database for Kafka users to quantify the loss of their topics if any. Apache License 2.0

• Cruise Control(Linkedin): Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a kafka cluster.It provides great value to Kafka users by simplifying the operation of Kafka clusters. BSD 2-Clause "Simplified" License

• Doctor Kafka(Pinterest): DoctorKafka is a service for Kafka cluster auto healing and workload balancing. Apache License 2.0

• kafka-kit (Datadog): Kafka data mapping and recovery tools. Apache License 2.0

• Kafka Monitor(Linkedin): Xinfra Monitor monitors the availability of Kafka clusters by producing synthetic workloads using end-to-end pipelines to obtain derived vital statistics - E2E latency, service availability & message loss rate.It reassigns partition & trigger preferred leader election to ensure each broker acts as leader of at least 1 partition of monitor topic. Apache License 2.0

• kafka-monitoring-stack-docker-compose: Demonstration on how to monitor Kafka using Prometheus and Grafana. Apache License 2.0

• KMinion (CloudHut): KMinion is a feature rich Prometheus exporter for consumer lag and cluster monitoring. MIT License

Test

• Kafka Junit: This library wraps Kafka’s embedded test cluster, allowing you to more easily create and run integration tests using JUnit against a "real" kafka server running within the context of your tests.No need to stand up an external kafka cluster! BSD 3-Clause "New" or "Revised" License

• kafka-streams-test-utils: Testing Kafka Streams.Apache License 2.0

• Mocked Streams: Scala DSL for Unit-Testing Processing Topologies in Kafka Streams. Apache License 2.0

• Test Container for Apache Kafka: Testcontainers can be used to automatically instantiate and manage Apache Kafka containers.More precisely Testcontainers uses the official Docker images for Confluent OSS Platform. MIT License

UI/Cluster Management

• AKHQ: Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more. Apache License 2.0

• CMAK: CMAK (previously known as Kafka Manager) is a tool for managing Apache Kafka clusters.See below for details about the name change. Apache License 2.0

• Kafdrop – Kafka Web UI: Kafdrop is a web UI for viewing Kafka topics and browsing consumer groups.The tool displays information such as brokers, topics, partitions, consumers, and lets you view messages. Apache License 2.0

• Kafka Eagle: A easy and high-performance monitoring system, as well as offsets or metadata and other kafka information. Apache License 2.0

• Kafkawize: Kafkawize : A Self service Apache Kafka Topic Management tool/portal.A Web application which automates the process of creating and browsing Kafka topics, acls, schemas by introducing roles/authorizations to users of various teams of an org. Apache License 2.0

• klustr: Monitoring tool and graphic visualizer for Apache Kafka that helps you track key metrics about your cluster in real-time. MIT License

• Remora: Kafka consumer lag-checking application for monitoring, written in Scala and Akka HTTP; a wrap around the Kafka consumer group command.Integrations with Cloudwatch and Datadog.Authentication recently added. MIT License

• Kowl: Kowl is a modern Kafka Web UI for exploring Kafka messages, configurations and more with a focus on a good UI/UX. Written in Go & React. Apache 2.0 License

Other Awesome Lists about Apache Kafka & Data Engineering

• Awesome Open-Source Data Engineering

• Awesome Data Engineering

• infoslack/awesome-kafka: This list is for anyone wishing to learn about Apache Kafka, but do not have a starting point.

• dharmeshkakadia/awesome-kafka: Everything about Apache Kafka

• Apache Kafka Ecosystem

License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.