seilylook / Cryptocurrency-Data-Pipeline Public

Notifications You must be signed in to change notification settings
Fork 0
Star 0

This project involved the integration of several cutting-edge technologies to ensure efficient data processing and management for Cryptocurrency analytics.

Apache-2.0 license

0 stars 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 6 Commits
docker		docker
mnt		mnt
.Python		.Python
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
pyvenv.cfg		pyvenv.cfg
reset.sh		reset.sh
restart.sh		restart.sh
start.sh		start.sh
stop.sh		stop.sh

Repository files navigation

Flow chart of the Cryptocurrency Data Pipeline

Check availability of Cryptocurrency cost in CoinMarketCap
Download top 10th Cryptocurrency(ex. BTC, ETH) cost with python
Save the Cryptocurrency cost in HDFS
Create a hive table to store Cryptocurrency cost from the HDFS
Process Cryptocurrency cost with Spark
Send an email notificatoin

About

This project involved the integration of several cutting-edge technologies to ensure efficient data processing and management for Cryptocurrency analytics.

airflow spark hive postgresql hdfs hue livy crpyto

Apache-2.0 license

Report repository

Releases

No releases published

Packages

No packages published

Languages