full-upserts

Full Upserts

In this recipe we'll learn how to use upsert functionality.

Pinot Version	1.0.0
Schema	config/orders_schema.json
Table Config	config/orders_table.json

This is the code for the following recipe: https://dev.startree.ai/docs/pinot/recipes/upserts-full

---

git clone git@github.com:startreedata/pinot-recipes.git
cd pinot-recipes/recipes/full-upserts

Spin up a Pinot cluster using Docker Compose:

docker compose \
		-f ../pinot-compose.yml \
		-f ../kafka-compose.yml up -d

Add table and schema:

docker cp config/orders_schema.json pinot-controller:/opt/pinot/
docker cp config/orders_table.json pinot-controller:/opt/pinot/

docker exec pinot-controller ./bin/pinot-admin.sh AddTable \
     -schemaFile /opt/pinot/orders_schema.json \
     -tableConfigFile /opt/pinot/orders_table.json \
    -exec

Open a tab to import messages into Kafka:

docker exec -it kafka kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders

Paste the following:

{"order_id":1,"customer_id":104,"order_status":"IN_TRANSIT","amount":29.35,"ts":"1632467063"}
{"order_id":2,"customer_id":105,"order_status":"COMPLETED","amount":3.24,"ts":"1618931459"}
{"order_id":3,"customer_id":103,"order_status":"OPEN","amount":9.77,"ts":"1626484196"}
{"order_id":4,"customer_id":104,"order_status":"COMPLETED","amount":90.35,"ts":"1623066325"}
{"order_id":5,"customer_id":105,"order_status":"OPEN","amount":55.52,"ts":"1635543905"}

Query Pinot:

select * 
from orders 
limit 10

Go back to the Kafka tab and paste the following:

{"order_id":5,"customer_id":105,"order_status":"CANCELLED","amount":55.52,"ts":"1635543948"}