please note that this repo was crearted to share an issue I'm facing. Hence the following instructions are to reproduce the issue. Until the issue ios resolved, you won't see data being moved
- Stand up a dockerised kafka cluster by running
docker-compose -f docker_kafka_server/docker-compose.yml up -d --build- Build an image for a dockerised spark server
docker build -f ./docker_spark_server/Dockerfile -t spark3.0.1-scala2.12-hadoop3.2.1 ./docker_spark_server- Build the spark application
docker run -e MAVEN_OPTS="-Xmx1024M -Xss128M -XX:MetaspaceSize=512M -XX:MaxMetaspaceSize=1024M -XX:+CMSClassUnloadingEnabled" --rm -v "${PWD}":/usr/src/mymaven -v "${HOME
}/.m2":/root/.m2 -w /usr/src/mymaven maven:3.6.3-jdk-8 mvn clean installRun the Spark application using the dockerised spark server
docker run -v $(pwd):/core -w /core -it --rm --network docker_kafka_server_default spark3.0.1-scala2.12-hadoop3.2.1:latest spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1 --deploy-mode client --class org.example.App target/simplespark-1.0-SNAPSHOT-jar-with-dependencies.jar- clean up dockerised kafka cluster by running
docker-compose -f docker_kafka_server/docker-compose.yml down