This project implements a Lakehouse Medallion Architecture using modern Data Stack tools such as Fivetran, Snowflake and dbt. The ficticious organization is an e-commerce company.

Extract data from many databases of Labor, Invalids and Social Affairs sectors and convert to appropriate structure and format, then upload to shared data warehouse and data mart. Thanks to that, people of state agencies can easily retrieve and analyze data based on the compiled data warehouse.

Tracking Tweet sentiment at scale using a pretrained Huggingface transformer (classifier) Model

This repository contains a comprehensive end-to-end Azure Data Engineering solution, covering the entire data lifecycle from ingestion to reporting. The project utilizes various Azure services and tools to achieve efficient data handling, transformation, and reporting.

Implemented enhanced analytics for trip transactions and ride-based data from SQL Server using Medallion Architecture with Azure Data Factory and Databricks.

Além de explorar boas práticas em python, o propósito é de aplicar conceitos de Arquitetura Medallion, programação orientada a objetos e ETL, utilizando dados públicos sociais dos municípios brasileiros. Este repositório utiliza ainda DuckDB e Streamlit para exibição dos resultados

In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our cloud provider.

Data Warehouse with ELT pipelines

Building a Data Lakehouse using the Medallion architecture.

Developed an end-to-end real time sentiment analysis system using Delta Lake Medallion architecture, with Hugging Face Transformer Model, Spark, and MLflow integration.