WeatherForecastPipeline-Snowflake

Weather Forecast Pipeline using snowflake and AWS (Lambda / S3 / Cloudwatch)

Pipeline Flow

1. Lambda Function triggered by event Bridge fetching Data from WeatherAPI

2. Json Data loads to S3 incrementally everyday

3. Event notification informs Snowpipe via an SQS queue when files are ready to load.

4. Snowpipe Auto ingest the data into varaint table using copy into from external stage.

Stage :

Variant table:

5. Next varaint stream will capture the new inserted data into varaint table and trigger the task that runs the store procedure to load data to Flat table. This layer is where all data is integrated and stored in flatten format

6. Again Flat stream will capture the new inserted data into flat table and trigger the task that runs the store procedure to load data to Target table along with proper data types and business validation and checks.

Note : Target table has SCD type 2 where only latest forecast data will be active.