ksummit-operator-demo

Kafka Summit NYC: Confluent Operator Demo aka Escape from EKS

Table of content

• Prerequisites
  • Install AWS IAM Authenticator
  • Install external DNS GCP
• Install external DNS EKS (Route 53)
  • Create certificate (this is require, one time-thing)
  • Validate DNS is setup
  • Install CP
  • Validate Kafka DNS Endpoints
    • AWS
    • GCP
• Applications
  • Create topics
  • Produce
  • Consume
  • Run standard script to bulk produce
  • Run standard script to bulk consume
  • Replicator

A demo from Kafka Summit NYC https://speaking.gamov.io/eZuDRU/kafka-on-kubernetes-does-it-really-have-to-be-the-hard-way

Prerequisites

1. GCP cluster name: demo-test

2. AWS cluster: eks-test

Install AWS IAM Authenticator

1. Download https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/0.4.0-alpha.1/aws-iam-authenticator_0.4.0-alpha.1_darwin_amd64

2. Move `mv <Download_path>/aws-iam-authenticator_0.4.0-alpha.1_darwin_amd64 aws-iam-authenticator

3. configure below on the .kube/config path

   aws eks --region us-west-2 update-kubeconfig --name eks-test

Install external DNS GCP

https://github.com/kubernetes-incubator/external-dns/blob/master/docs/tutorials/gke.md

export VALUE_PATH=/Users/viktor/workspace/k-demo
helm install  -f ${VALUE_PATH}/gcp/gcp.yaml --name external-dns --namespace kube-system --set externaldns.enabled=true --set global.externaldns.user=platform-semaphore@platform-eng-204922.iam.gserviceaccount.com ./confluent-operator

Install external DNS EKS (Route 53)

Create IAM Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "route53:ListHostedZones",
                "route53:ListResourceRecordSets"
            ],
            "Resource": ["_"]
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:ChangeResourceRecordSets"
            ],
            "Resource": ["_"]
        }
    ]
}

export VALUE_PATH=/Users/viktor/projects/k-demo

helm install -f $\{VALUE_PATH}/aws/aws.yaml --name external-dns --namespace kube-system --set externaldns.enabled=true ./confluent-operator

Create certificate (this is require, one time-thing)

./client gcp
./client aws

Validate DNS is setup

Check Route54 or GCP DNS, if configurations are not there delete the external-dns pod running on the kubesystem (sometime it takes more time)

Install CP

add location of demo Helm values files $PATH

export VALUE_PATH=/Users/viktor/projects/k-demo

# AWS:
    scripts/deployCP.sh -c operator -p aws -f $\{VALUE_PATH}/aws/aws.yaml --debug
    scripts/deployCP.sh -c zookeeper -p aws -f $\{VALUE_PATH}/aws/aws.yaml  --debug
    scripts/deployCP.sh -c kafka -p aws -f $\{VALUE_PATH}/aws/aws.yaml  --debug
    scripts/deployCP.sh -c controlcenter -p aws -f $\{VALUE_PATH}/aws/aws.yaml  --debug

# GCP:

 scripts/deployCP.sh -c zookeeper -p gcp -f ${VALUE_PATH}/gcp/gcp.yaml  --debug
 scripts/deployCP.sh -c kafka -p gcp -f ${VALUE_PATH}/gcp/gcp.yaml  --debug
 scripts/deployCP.sh -c replicator -p gcp -f ${VALUE_PATH}/gcp/gcp.yaml --debug
 scripts/deployCP.sh -c controlcenter -p gcp -f ${VALUE_PATH}/gcp/gcp.yaml  --debug
 ## scripts/deployCP.sh -c all -p gcp -f ${VALUE_PATH}/gcp/gcp.yaml  --debug

Validate Kafka DNS Endpoints

$\{VALUE_PATH}/k-demo/scripts/validateKafkaCluster.sh operator kafka    #(1)

1. use dig command to validate if the DNS is resolved (during demo, you should deploy before hand otherwise the resolve will take time.)

AWS

kafka-broker-api-versions --command-config aws/kafka.properties --bootstrap-server kafka.platformops.aws.devel.cpdev.cloud:9092

GCP

kafka-broker-api-versions --command-config gcp/kafka.properties --bootstrap-server kafka-nyc.platformops.dev.gcp.devel.cpdev.cloud:9092`

Applications

Create topics

• GCP

  ccloud -c gcp/ topic create example --replication-factor 3

• AWS

  ccloud -c aws/ topic create example --replication-factor 3

Produce

• GCP:

  ccloud -c gcp/ produce -t example

• AWS:

  ccloud -c aws/ produce -t example

Consume

• GCP

  ccloud -c gcp/ consume -t example -b

• AWS

  ccloud -c aws/ consume -t example -b

Run standard script to bulk produce

• GCP

  seq 10000 | kafka-console-producer --topic example --broker-list kafka.platformops.dev.gcp.devel.cpdev.cloud:9092 --producer.config gcp/kafka.properties

• AWS

  seq 10000 | kafka-console-producer --topic example --broker-list kafka.platformops.aws.devel.cpdev.cloud:9092 --producer.config aws/kafka.properties

Run standard script to bulk consume

• GCP:

  kafka-console-consumer --from-beginning --topic example --bootstrap-server kafka.platformops.dev.gcp.devel.cpdev.cloud:9092 -consumer.config gcp/kafka.properties

• AWS:

  kafka-console-consumer --from-beginning --topic example --bootstrap-server kafka.platformops.aws.devel.cpdev.cloud:9092 -consumer.config aws/consumer.properties

Replicator

kubectl port-forward replicator-0 8083:8083

kpf replicator-0 8083:8083

curl -s -XGET http://localhost:8083/connector-plugins| jq '.[].class'

curl -XPOST -H "Content-Type: application/json" --data @replicator.json http://localhost:8083/connectors