This project provides a simulated real-time visualization of the energy production of the main countries of the Euro-Asian continent and their CO2 emissions, the technologies used are the following:
- Docker to create and manage containers
- Electricity Map as real time data source
- Logstash for data ingestion
- ZooKeeper + Kafka for data stream processing
- Spark to process data
- Elastic Search for data indexing
- Kibana for data visualization
$ git clone https://github.com/
$ cd
$ docker-compose up Elasticsearch:
- user: elastic
- password: energyanalyzer
Kibana:
- user: kibana_system
- password: energyanalyzer
Once the Dashboard has been built, it can be saved together with all the inserted objects (views and maps) in an ndjson type file. To do this, click on the menu on the left and go to Management> Stack Management, in the drop-down on the left click on Kibana> Saved Objects, find your Dashboard and export it, making sure to include the objects inside it.
To reload the saved Dashboard re-enter the Saved Objects section and click on import.
The elasticsearch01 container could come out with exit status 78, going to see the errors you will probably see the message
- "Elasticsearch: Max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]".
The error message states that the memory granted to the WSL is too low
If this is the case, it should be sufficient to run these two commands using a prompt before compose up:
$ wsl -d docker-desktop
$ sysctl -w vm.max_map_count=262144| Service | Link | Note |
|---|---|---|
| KafkaUI | http://localhost:8080 | To check the status of topics and their messages |
| Cluster Elastic Search | https://localhost:9200/ | To view the ES index |
| Kibana | http://localhost:5601/ | To access the dashboard |
- Gabriele Sanguedolce - Francesco Cristoforo Conti