prometheus-exemplar

A random duration response app to test Grafana LGTM stack.

Installation

• git clone code

git clone git@github.com:grafanafans/prometheus-exemplar.git
cd prometheus-exemplar

• start app

docker-compose up -d

stop app

docker-compose down

How to test

use wrk to send requests

wrk http://localhost:8080/v1/books
wrk http://localhost:8080/v1/books/1

Then visit http://localhost:3000 page to see demo app dashboard.

All in one with Grafana

Add data sources

add Mimir

Add exemplar configuration:

Notes:

• HTTP URL：http://lb.com/metrics

add Tempo

• HTTP URL:http://lb.com/traces

add Loki with Derived fields

• HTTP URL:http://lb.com/logs
• Regex:(?:traceID|trace_id|TraceID|TraceId)=(\w+)
• NOTICE: Regex can be modified according to your own TraceID characters.

Query exemplar metrics

query mimir with tempo

In "Explore" module, when you query metrics by Mimir:

histogram_quantile(0.95, rate(http_durations_histogram_seconds_bucket{}[1m]))

then open the "Exemplars" flag, it shows a green exemplar point, then click "Query with Tempo" to jump to Tempo Explore.

query loki with tempo

{app="exemplar-demo"} |= `traceID`  

click "Tempo" to jump to Tempo Explore.