Pattern: Predictive Caching

Quick Start

Download required data sources from Kaggle into the datasource/ directory and prepare the datasets for their usage. The required steps are described in the Datasources Readme file.

Build container images:

./gradlew clean build bootBuildImage

Provide read permissions to those files to anyone to allow access within container:

chmod -R g+r,o+r datasource/
chmod -R g+r,o+r news-db/
chmod -R g+r,o+r monitoring/

If you first want to pull all images, run the following command (not required):

docker-compose pull --ignore-pull-failures

Run demo environment:

docker-compose up -d

Wait until the user-interest-db-init has been completed:

docker ps -a --format 'table {{.Names}}\t{{.Status}}'

NAMES                         STATUS
interest-feed                 Up 16 seconds
user-interest-db-init         Exited (0) 10 seconds ago
news                          Up 11 seconds
prometheus                    Up 16 seconds
grafana                       Up 16 seconds
mariadb-prometheus-exporter   Up 16 seconds
news-db                       Up 17 seconds
user-interest-db              Up 16 seconds
redis-prometheus-exporter     Up 16 seconds
node_exporter                 Up 17 seconds
cadvisor                      Up 17 seconds

Hint: The setup only works with Docker and has only been tested on Linux.

Websites:

• Grafana Dashboards: http://localhost:3000/dashboards

• Prometheus Targets: http://localhost:9090/targets

News Service

Test curl request to get recommended news:

curl -s -XGET -H "Accept: application/json" 'http://localhost:8081/news/recommended?topics=HOME%20%26%20LIVING,COMEDY&fromDate=2013-07-29&untilDate=2013-08-05&limit=20'

Hint: In a previous version we used a POST request for the /recommendedNews endpoint to simplify specification of parameters. However, Http client-side cache is not possible for POST requests. Therefore, we switched to a GET request.

Test curl request to get latest news:

curl -s -XGET -H "Accept: application/json" 'http://localhost:8081/news/latest?untilDate=2013-08-05&limit=3'

Interest Feed Service

Test curl request to get feed for a user:

curl -s -XGET -H "Accept: application/json" 'http://localhost:8082/feed/mariameyer000?date=2013-08-05'

Test curl request to get usernames:

curl -s -XGET -H "Accept: application/json" 'http://localhost:8082/usernames?limit=10'

Test curl request to get interests:

curl -s -XGET -H "Accept: application/json" 'http://localhost:8082/interests?limit=10'

Test curl request to get interests by user:

curl -s -XGET -H "Accept: application/json" 'http://localhost:8082/interests/mariameyer000'

ToDos

• ✓ Feed should always contain news (use latest feeds API)

• ✓ API to show all users (and maybe their interests)

• ✓ Benchmark app

• ✓ Monitoring setup and metrics for CPU time, CPU usage and memory of services and dbs

• ✓ Metrics for service and db internals like http connections, …​

• ❏ UI for Feeds and Usernames

• ❏ Implementation of simple Feed caching (enable via config)

  • ✓ Http client cache in Interest-Feed service for News service endpoints

  • ✓ Http client cache in Benchmark for Interest-Feed service endpoints

  • ❏ Redis client-side cache in Interest-Feed service

  • ❏ MariaDB client-side cache in News service

  • ❏ Http independent cache for Interest-Feed and News service

• ❏ Monitoring for Watt and CO2e

• ❏ Implementation of predictive caching

• ❏ Setup for Kubernetes

• ❏ Documentation with images