I am using confluent Kafka cluster to produce and consume scraped data. In this project, I've created a real-time data pipeline that utilizes Kafka to scrape, process, and load data onto S3 in JSON format. With a producer-consumer architecture, I ensure that the data is in the right format for loading onto S3 by performing minor transformations

Data Engineering project using data streaming produced by python applications, ETL process and availability for ad-hoc SQL queries in the AWS cloud

Data Streaming and Batch processing using AWS Services