This repository contains my submission for the Final Project: Data Analysis using Spark. The original files were provided by the IBM Skills Network as part of the Introduction to Big Data with Spark and Hadoop course on Coursera. I have made modifications to fulfill the project requirements.
-
You are welcome to use this repository as a reference or starting point for your own project.
-
If you choose to fork this repository, please ensure that you comply with the terms of the Apache License and give proper credit to the original authors.
As a data engineer, I’ve been assigned by our HR department to design a robust data pipeline capable of ingesting employee data in CSV format. My responsibilities include analyzing the data, implementing necessary transformations, and enabling the extraction of valuable insights from the processed data.
- Create a DataFrame from a CSV file
- Define a schema for the data
- Perform transformations and actions using Spark SQL
Install the required libraries using the provided requirements.txt file. The command syntax is:
python3 -m pip install -r requirements.txtDownload the required employees csv file using the terminal command:
wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-BD0225EN-SkillsNetwork/data/employees.csv- IBM Skills Network © IBM Corporation 2023. All rights reserved.