Skip to content

Docker container for Kafka and Schema Registry

Jump to bottom
authorjapps edited this page Jun 13, 2019 · 6 revisions 
 
After following this page you should be successfully able to bring up a dockerized 
Kafka broker and run the tests.
                                         (Reading 1st time: 5min, Afterwards: 2min)

Table Of Content

(Follow the instructions from here to bring up a Single Node Kafka).

How to configure/create a Docker compose file?

See here the docker-compose file

How to bring down the container using the compose file ?

  • $ docker-compose -f kafka-schema-registry.yml down

If the containers are up, the above command will bring them down and if they are not up, then it does no harm.

How to bring up the container using the compose file ?

  • Clone this repo

  • Go to the zerocode-docker-factory/compose dir in a terminal window

  • Then run

    • $ docker-compose -f kafka-schema-registry.yml up -d

    or

    • $ docker-compose -f kafka-schema-registry.yml up

This brings up both Zookeeper and Kafka server(server and broker mean the same thing on this page)

You should see the following here with "done" in green color.

$ docker-compose -f kafka-schema-registry.yml up -d
Creating network "compose_default" with the default driver
Creating compose_zookeeper_1 ... done
Creating compose_kafka_1     ... done
Creating compose_schema-registry_1 ... done
Creating compose_ksql-server_1     ... done
Creating compose_rest-proxy_1      ... done
Creating compose_ksql-cli_1        ... done
$

How to check the status of Kafka and Zookeeper?

$ docker ps
$ docker ps
CONTAINER ID        IMAGE                                   COMMAND                  CREATED              STATUS              PORTS                          NAMES
fd9ab34e500e        confluentinc/cp-ksql-cli:5.1.0          "/bin/sh"                About a minute ago   Up About a minute                                  compose_ksql-cli_1
d3f3e75a2832        confluentinc/cp-ksql-server:5.1.0       "/etc/confluent/dock…"   About a minute ago   Up About a minute   0.0.0.0:8088->8088/tcp         compose_ksql-server_1
c0266d234c06        confluentinc/cp-kafka-rest:5.1.0        "/etc/confluent/dock…"   About a minute ago   Up About a minute   0.0.0.0:8082->8082/tcp         compose_rest-proxy_1
74034ec1e98c        confluentinc/cp-schema-registry:5.1.0   "/etc/confluent/dock…"   About a minute ago   Up About a minute   0.0.0.0:8081->8081/tcp         compose_schema-registry_1
6ec5fdb06e24        confluentinc/cp-kafka:5.1.0             "/etc/confluent/dock…"   About a minute ago   Up About a minute   0.0.0.0:9092->9092/tcp         compose_kafka_1
9749a04c41a0        confluentinc/cp-zookeeper:5.1.0         "/etc/confluent/dock…"   About a minute ago   Up About a minute   2181/tcp, 2888/tcp, 3888/tcp   compose_zookeeper_1
$ $ 

Note-
See under the "NAMES" column(right hand ride) above, for `compose_kafka_1` as container name
and the "PORTS" column to see which port the container is listening to for the external listners.

How to verify the port has been exposed outside docker container ?

  • To make sure Kafka broker is running and it has exposed the port 9092 to external client outside the docker,

    • nc -vz localhost 9092

$ nc -vz localhost 9092   
found 0 associations
found 1 connections:
     1: flags=82<CONNECTED,PREFERRED>
        outif lo0
        src ::1 port 51288
        dst ::1 port 9092
        rank info not available
        TCP aux info available

Connection to localhost port 9092 [tcp/XmlIpcRegSvc] succeeded!

Note- If a port is not exposed, you get the following message.
--------------------------------------------------------------
$ nc -vz localhost 9093  
nc: connectx to localhost port 9093 (tcp) failed: Connection refused
nc: connectx to localhost port 9093 (tcp) failed: Connection refused
$

How to verify that a message can be produced to and consumed from a topic?

Get into the container

$ docker exec -it compose_kafka_1 bash
root@9b5001dc2931:/#

For producing a message to a topic(named test)

root@9b5001dc2931:/# echo "test-XYZ" | kafka-console-producer --broker-list kafka:29092 --topic test
>>root@9b5001dc2931:/# 
Note-
If you see no error means, it might been produced correctly

For consuming a message from a topic(named test)

root@9b5001dc2931:/# kafka-console-consumer --bootstrap-server kafka:29092 --topic test --from-beginning
test-XYZ

Press Ctrl+c to stop this process and come to the command prompt

Now, how to run the Zerocode tests?

In your kafka_server.properties upddate the broker host to kafka.bootstrap.servers=localhost:9092 and run the Producer and Consumer tests.

HelloWorld tests are here

How to bring down the containers once you are done with your testing?

Already explained in the above section, see here

Clone this wiki locally