The goal of this workshop is to provide an overview of the Kafka architecture, fundamental concepts and basic architecture patterns and use cases for Kafka.
Additionally, we will introduce Spring Boot and the Spring Kafka libraries and demonstrate their use over several progressive labs demonstrating the practices of producing, consuming and streaming events to and from Kafka.
Apache Kafka is a distributed data store optimized for ingesting and processing streaming data in real-time. Streaming data is data that is continuously generated by thousands of data sources, which typically send the data records in simultaneously. A streaming platform needs to handle this constant influx of data, and process the data sequentially and incrementally.
Kafka provides three main functions to its users:
- Publish and subscribe to streams of records
- Effectively store streams of records in the order in which records were generated
- Process streams of records in real time
Kafka is primarily used to build real-time streaming data pipelines and applications that adapt to the data streams. It combines messaging, storage, and stream processing to allow storage and analysis of both historical and real-time data.
- Developed at LinkedIn in and open sourced in 2011
- Graduation from the Apache Incubator occurred on 23 October 2012
- Confluent formed in 2014
- Streams API released in 2016
Apache Kafka is licensed under the Apache 2.0 license which is a popular and permissive open source license.
Confluent Community License applies to any pre-built connectors, Schema Registry, REST Proxy and ksqlDB.
- MSK
- Kafka in Kubernetes
- Kubernetes Operators