Near real-time Campaign reporting

How to acheive new real-time campaign reporting using Aerospike and Kafka Read the articles at Medium

Javascript and Node.js is used in each service although the same solution is possible in any language

The solution consists of:

• A Publisher Simulator - Node.js
• An Event Collector service - Node.js
• An Aggregator/Reducer servic - Node.js
• A Campaign Service - Node.js and Apollo Server
• A Campaign UI - React, Material UI and Apollo Client
• Aerospike configurations enabling Kafka
• Docker compose yml
• Docker containers from DockerHub for:
  • Aerospike Enterprise Edition (https://dockr.ly/2KZ6EUH)
  • Zookeeper (https://dockr.ly/2KZgaaw)
  • Kafka (https://dockr.ly/2L4NwVA)
  • Kafka CLI (https://dockr.ly/2KXYEn4)

Companion Code

This example uses Docker and Docker Compose to create a simple solution for near real-time Ad Campaign reporting on a fixed set of campaign dimensions. The solution presented in this series relies on Kafka, Aerospike's edge-to-core data pipeline technology, and Apollo GraphQL

Setup

Clone the GitHub repository with

$ git clone https://github.com/helipilot50/real-time-reporting-aerospike-kafka 
$ cd real-time-reporting-aerospike-kafka

Edit the file docker-compose.yml and add you Aerospike customer user name and password

  edge-exporter:
    container_name: edge_exporter
    build:
      context: ./edge-exporter
      args:
        AEROSPIKE_USER_NAME: "<your user name>"
        AEROSPIKE_PASSWORD: "<your password>"

Copy your Feature Key File to project etc directory for the Edge and Core datastore; and to the edge exporter

$ cp ~/features.conf <project-root>/aerospike/edge/etc/aerospike/features.conf
$ cp ~/features.conf <project-root>/aerospike/core/etc/aerospike/features.conf
$ cp ~/features.conf <project-root>/edge-exporter/features.conf

Then run

$ docker-compose up

Reseting the data

The data in the two Aerospike clusters and the two Kafka topics are stored locally on the host machine. The data can be reset (deleted) by running this bash script:

$ ./delete-data.sh                                                                                                                         

Disclaimer

The code and the example solution are entirely my own work and not endorsed by Aerospike, Confluent or Apollo. The code is PoC quality only and it is not production strength, and is available to anyone under the MIT License.