Kafka Stress - Stress Test Tool for Kafka Clusters, Producers and Consumers Tunning

Introduction

Kafka Stress is a simple CLI Tool to produce and consume events on Kafka topics in high throughput to tests and validate brokers, topics, partitions and consumers performances.

Installation

Docker

docker pull fidelissauro/kafka-stress:latest

docker run --network host -it fidelissauro/kafka-stress:latest --bootstrap-servers 0.0.0.0:9092 --topic kafka-stress --test-mode consumer --consumers 6

MacOS amd64

wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.9/kafka-stress_0.0.9_darwin_amd64 -O kafka-stress 
mv kafka-stress /usr/local/bin 
chmod +x /usr/local/bin/kafka-stress

MacOS arm64

wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.9/kafka-stress_0.0.9_darwin_arm64 -O kafka-stress 
mv kafka-stress /usr/local/bin 
chmod +x /usr/local/bin/kafka-stress

Linux amd64

wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.9/kafka-stress_0.0.9_linux_amd64 -O kafka-stress 
mv kafka-stress /usr/local/bin 
chmod +x /usr/local/bin/kafka-stress

Linux arm64

wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.9/kafka-stress_0.0.9_linux_arm64 -O kafka-stress 
mv kafka-stress /usr/local/bin 
chmod +x /usr/local/bin/kafka-stress

Freebsd amd64

wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.9/kafka-stress_0.0.9_freebsd_amd64 -O kafka-stress 
mv kafka-stress /usr/local/bin 
chmod +x /usr/local/bin/kafka-stress

Freebsd arm64

wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.9/kafka-stress_0.0.9_freebsd_arm64 -O kafka-stress 
mv kafka-stress /usr/local/bin 
chmod +x /usr/local/bin/kafka-stress

v0 Usage

Usage of kafka-stress:
  -ack int
    	Required ACKs to produce messages (default 1)
  -balancer string
    	Balance algorithm for producer mode; Ex: hash,murmur2,crc32 (default "hash")
  -batch-size int
    	Batch size for producer mode
  -bootstrap-servers string
    	Kafka Bootstrap Servers Broker Lists (default "0.0.0.0:9092")
  -consumer-group string
    	Consumer group name (default "kafka-stress")
  -consumers int
    	Number of consumers will be used in topic (default 1)
  -create-topic
    	Auto Create Topic?
  -events int
    	Numer of events will be created in topic (default 10000)
  -format string
    	Events Format; ex string,json,avro (default "string")
  -schema string
    	Schema
  -schema-registry string
    	Schema Registry URL (default "0.0.0.0:8081")
  -size int
    	Message size in bytes (default 62)
  -ssl-enabled
    	SSL Mode
  -test-mode string
    	Test Type; Ex producer;consumer. Default: producer (default "producer")
  -topic string
    	Kafka Stress Topics (default "kafka-stress")
  -verbose
    	Verbose Mode; It Prints Events consumed
  -zookeeper-servers string
    	Zookeeper Connection String (default "0.0.0.0:2181")

Producer

kafka-stress --bootstrap-servers localhost:29092 --events 30000 --topic kafka-stress

kafka-stress --bootstrap-servers localhost:29092 --events 10000 --topic kafka-stress

Sent 10000 messages to topic kafka-stress with 0 errors
Tests finished in 1.232463918s. Producer mean time 8113.83/s

Customize ACK's

Use --ack parameter to customize ack's quorum

kafka-stress --bootstrap-servers 0.0.0.0:9092 --events 20000 --topic tunning-3 --test-mode producer --ack 1

Producer Format

You can produce random data in string and json format using --format flag

kafka-stress --bootstrap-servers 0.0.0.0:9092 --events 20000 --topic tunning-3 --test-mode producer --ack 1 --format json

Producer Balance Algorithms

You can specify some algorithms to balance producer messages like hash, murmur2 and crc21 using --balance flag

kafka-stress --bootstrap-servers 0.0.0.0:9092 --events 20000 --topic tunning-3 --test-mode producer --balance murmur2 --format json

Consumer

kafka-stress --bootstrap-servers 0.0.0.0:9092 --topic kafka-stress --test-mode consumer --consumers 6

```bash
kafka-stress --bootstrap-servers 0.0.0.0:9092 --events 10000 --topic kafka-stress --test-mode consumer  --consumers 6

...
[Consumer 6] Message from consumer group kafka-stress at topic/partition/offset kafka-stress/1/0: 65b27cc0-1053-4fbc-b7a9-a40972fcaca6 = a124e95d-9226-4eb9-9169-411318bb6e4e
[Consumer 5] Message from consumer group kafka-stress at topic/partition/offset kafka-stress/2/0: 35fc905f-27f3-4b9f-81db-af89c1fa4c28 = 94f06092-589c-4030-8e3f-66a5a980123b
...

Customize consumer-group name

Use --consumer-group to change customer group name used by workers.

kafka-stress --bootstrap-servers 0.0.0.0:9092 --topic kafka-stress --test-mode consumer --consumer-group custom-consumer-group

Verbose mode

Use --verbose to print keys and events

kafka-stress --bootstrap-servers 0.0.0.0:9092 --topic kafka-stress --test-mode consumer --verbose

Authentication methods

SSL

Use --ssl-enabled to enable ssl authentication.

kafka-stress --bootstrap-servers 0.0.0.0:9092 --topic kafka-stress --test-mode consumer --ssl-enabled

Roadmap

Improve reports output

• Add retry mechanism
• Add message size options
• Add test header
• Add SCRAM authentication
• Add SASL authetication
• Add TLS authetication
• Add IAM authentication for Amazon MSK
• Add Unit Tests
• Add time based tests
• Add Schema Registry / AVRO, JSON, PROTO Support