3 Days Kafka Training Course

Prerequisites

- Knowledge of Java
- Knowledge of Bash
- Download Confluent Plaform

Slide

Topics

- Publish Subscribe
- Kafka Concepts
    - Topics
    - Producers
    - Consumers
- Kafka System
    - Brokers
    - Partitioning
    - Serialization
- Kafka Internals
    - Commit Log
    - Log Compaction
    - Fault Tolerance
        - Broker via Replication
        - Producer via Transactionality

TODO

[ ] Exercise 14 Join KGlobal Table [ ] Exercise X on Join Stream-Stream [ ] Exercise X on Session Windows [ ] KSQL in Java [ ] Exercise 17-18 Readme

Day 1 Kafka Basics: List of Exercises

• ABC

  • Exercise 1: Console 1
  • Exercise 2: Console 2

• Advanced

  • Exercise 3: Producer
  • Exercise 4: Consumer
  • Exercise 5: P/C
    • Serialization
    • Deserialization
    • Partitioning

• Expert

  • Exercise 6: Word Count P/C
  • Exercise 7: Offset Management
  • Exercise 8: Homework

Day 2 Kafka Stream: List of Exercises

• Exercise 10: Pipes
• Exercise 11: Word Count
  • Processor
  • Kafka Stream
• Exercise 12: Rolling Average
• Exercise 13: Time-Based Window
  • Processing Time
  • Event Time
• Exercise 14: Stream Enrichment
• Exercise 15: TBD
• Exercise 16: Homework
• Exercise 17: Pageviews

Day 3 KSQL: List of Exercises

TBD

Running Kafka (Requires Linux)

Download Confluent Platform here

enter the bin folder and verify if all the scrits are executable

#grant permission
chmod +x *.sh

ssh to the virtual machine with port forwarding

ssh -p 3022 -L 9092:localhost:9092 -L 2081:localhost:2081 tartu@localhost

Then start zookeeper. It's address is localhost:2181

bin/zookeeper-server-start etc/kafka/zookeeper.properties

Then we start a kafka broker

bin/kafka-server-start etc/kafka/server.properties

If you want to start a second broker you MUST change the ID in the configuration and the port as indicated below. Suggestion, also differentiate the log folder.

etc/kafka//server-1.properties:
    broker.id=1
    listeners=PLAINTEXT://:9092
    log.dirs=/tmp/kafka-logs-1
    

etc/kafka//server-2.properties:
    broker.id=2
    listeners=PLAINTEXT://:9093
    log.dirs=/tmp/kafka-logs-2

Utilities

Topology-Viz

CLI

Create topics from the CLI

bin/kafka-topics --create --zookeeper localhost:2181 --replication-factor X --partitions Y --topic <name>

List existing topics

bin/kafka-topics --list --zookeeper localhost:2181

Describe a certain topic

bin/kafka-topics --describe --zookeeper localhost:2181 test

Output

Topic: test PartitionCount: 1 ReplicationFactor: 1 Configs: Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0 Offline:

Other Utilities

bin/kafka-console-producer --broker-list localhost:9092 --topic test

bin/kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning

bin/kafka-topics --list --zookeeper localhost:2181

bin/schema-registry-start ./etc/schema-registry/schema-registry.properties