This Project is a group work with Armin Ayatollahi, Zahra Habibzadeh Yara M. Bahram , mohammad-nili. in this project we used online Persian Twitter data for Designing a real-time system.
For analyzing the content of Persian tweets, we developed a complete data processing pipeline, starting with stream data collection, prepocessing and maintaining the data, sending it to Elastic Search, processing and storing hashtag/channel history in Cassandra, saving real-time statistics in Redis, Performing real-time forecasting with Spark, statistical analysis of data with Superset/Clickhouse, and visualizations using Flask, React, and Kibana front-ends. the centrality of the data transfer was with Kafka.
