sparkify-cassandra

This folder contains the necessary program files to create sparkifydb keyspace in cassandra , create and load tables that are optimized for some read queries.

Contents

1. etl.ipynb
   • Jupyter notebook file used to create the tables and load with data.
2. event_data
   • folder containing all the data files.
3. images
   • folder containing the images used in the ipynb file.
4. event_datafile_new.csv
   • file generated by the intermediate staging step in etl.
5. README.md
6. environment.yaml
   • conda environment file to import the python environment used by the project.

Installation

Setup Cassandra

Please follow the instructions given Here to install and run cassandra on Mac OS X. Once Cassandra is installed run below commands to start cassandra.

1. Start cassandra

   launchctl load ~/Library/LaunchAgents/homebrew.mxcl.cassandra.plist
   

2. Use below command to stop cassandra

   launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.cassandra.plist
   

Setup sparkify-cassandra project

1. Use the following command to clone the project repository.

   git clone https://github.com/shilpamadini/sparkify-cassandra.git
   

2. Navigate to project folder.Create the environment using below command.

   conda env create -f environment.yaml
   

3. Activate the conda environment

   source activate dand_py3
   

4. Run the following command to launch jupyter notebook.

   jupyter notebook
   

5. Open etl.ipynb and run the commands one after the other.

Functionality

A startup called Sparkify wants to analyze the data they've been collecting on songs and user activity on their new music streaming app. The analysis team is particularly interested in understanding what songs users are listening to. Currently, there is no easy way to query the data to generate the results, since the data reside in a directory of CSV files on user activity on the app. This project aims to create an Apache Cassandra database which can create queries on song play data to answer the questions

Cassandra tables are designed in such a way that data in the tables is denormalized and optimized for faster reads. Since we already know the queries that the analytics will be interested in each table to created to support one specific query.