Analysing Yelp Dataset using Spark and Comparative Study with different distributed processes

Dataset - ( ~9 GB )

https://www.kaggle.com/datasets/yelp-dataset/yelp-dataset

Tables used :

Yelp_academic_dataset_business.json

Yelp_academic_dataset_checkin.json

Yelp_academic_dataset_review.json

Yelp_academic_dataset_tip.json

Yelp_academic_dataset_user.json

---

Architecture Diagram

---

Spark Features Implementation

1. Persistence
2. Lazy evaluation
3. Fault tolerance
4. Data Partitioning
5. Parallelism
6. Transparency

1. Persistence

2. Lazy evaluation

Implementation of a Distributed System to execute Spark Using Multiple Computers (1 master and 2 workers)

¸

3. Fault tolerance

4. Parallelism

5. Data Partitioning

6. Transparency - Data Lineage

Ensuring transparency in Spark data processing with the explain() method

9GB file pyspark execution for the usecase in local system(8 Core MacOs)

Yelp Dataset Analysis & Comparative Analysis of Distributed Programming Techniques

https://public.tableau.com/app/profile/san.vinoth/viz/YelpDatasetComparativeAnalysis/YelpAnalysis?publish=yes

Additional Works - (Hadoop Cluster)

Project Components:

Learning

Spark

Databricks

Azure Blob

Tableau

Setting up standalone clusters

Hadoop environment setup

Team Work