End-to-end batch and streaming data pipeline on AWS to process user ratings and activity data. Leverages Amazon RDS, Glue, S3, Kinesis, and PostgreSQL with pgvector for real-time recommendation generation and model training.

This project is about building a serverless ETL pipeline using open-meteo weather API.

Streaming data pipeline to continuously load data from an Amazon MSK or MSK Serverless cluster to Amazon S3 using Amazon Kinesis Data Firehose.

Data Pipeline for CDC data from MySQL DB to Amazon OpenSearch Service through Amazon Kinesis using Amazon Data Migration Service(DMS).

Automated Functional Testing (AFT) packages supporting nodejs based Functional Testing

Describes the concepts of lambda architecture and the actual deployment process with an example of building a serverless business intelligence systems using Amazon Kinesis, S3, Athena, OpenSearch Service, and QuickSight.

Lightweight and simple Python package to quickly send JSON data to various AWS services.

A complete AWS pipeline that collects, compiles, preprocesses streaming data and trains an ML model

AWS Kinesis plugin for Filebeat

Subscribing to CloudWatch Logs with various destinations

Streaming data pipeline on aws, Tech session repository for hist

AWS kinesis data stream, kinesis data firehose, a transformer with lambda and store into s3 bucket

Realtime data pipeline

The data preprocessing code for FIREBALL

This Big Data project consists of obtaining data on vehicle theft in the city of São Paulo and consolidating it in a counting and heat map, in order to show areas with a higher index of this type of crime. All applicable in AWS Resources.

Big Data Lab | Stream Delivery AWS Kinesis Firehose

Ingestion of bid requests through Amazon Kinesis Firehose and Kinesis Data Analytics. Data lake storage with Amazon S3. Restitution with Amazon QuickSight and CloudWatch. 📉

❄️ Stream real-time data into Snowflake with Amazon Kinesis Firehose

Big data Text Classification: 4M records

A wrapper around AWS Kinesis Firehose with retry logic and custom queuing behavior. Requires node >= 6.0.0