Panels in some dashboards return no data points

[teutat3s]

Thank you for these beautiful grafana dashboards giving great insights into the inner workings of TDC.

After deploying latest cmon + triton-prometheus + triton-grafana I noticed some panels returning no data points (often in relation to moray).

Dashboard: CNAPI

• Panel CNAPI moray latency

histogram_quantile(0.90, sum(rate(fast_client_request_time_ms{service=\"cnapi\"}[5m])) by (le))
histogram_quantile(0.95, sum(rate(fast_client_request_time_ms{service=\"cnapi\"}[5m])) by (le))
histogram_quantile(0.99, sum(rate(fast_client_request_time_ms{service=\"cnapi\"}[5m])) by (le))

Dashboard Manatee

• All panels except Manatee{0,1,2} cpu|memory usage return no data points
• Moray Database Queries (rows/sec)

sum(rate(pg_stat_database_tup_deleted[5m]))
sum(rate(pg_stat_database_tup_inserted[5m]))
sum(rate(pg_stat_database_tup_updated[5m]))
sum(rate(pg_stat_database_tup_fetched[5m]))

• Disk Reads (per second)

sum(rate(pg_stat_database_blks_read[5m])) by (backend)
sum(rate(pg_stat_database_blks_hit[5m])) by (backend)

• User Tables Query Time (per second)

sum(rate(pg_stat_user_tables_querytime_ms[5m]) / 1000) by (backend)

• User Table Stats (per second)

sum(rate(pg_stat_user_tables_n_tup_ins[5m]))
sum(rate(pg_stat_user_tables_n_tup_del[5m]))
sum(rate(pg_stat_user_tables_n_tup_upd[5m]))
sum(rate(pg_stat_user_tables_n_tup_hot_upd[5m]))

• Connections to Manatee (moray DB)

pg_stat_activity_connections{state=\"active\",datname=\"moray\"}

• Manatee Transactions Committed (tx/s)

sum(rate(pg_stat_database_xact_commit[5m])) by (backend)

• Index scans per second by relname

rate(pg_stat_user_tables_idx_scan{relname!~\"napi_ips_.*\"}[5m])
sum(rate(pg_stat_user_tables_idx_scan{relname=~\"napi_ips_.*\"}[5m]))

• Tuples fetched by index scans per second by relname

rate(pg_stat_user_tables_idx_tup_fetch{relname!~\"napi_ips_.*\"}[5m])
sum(rate(pg_stat_user_tables_idx_tup_fetch{relname=~\"napi_ips_.*\"}[5m]))

• Sequential scans per second by relname

rate(pg_stat_user_tables_seq_scan{relname!~\"napi_ips_.*\"}[5m])
sum(rate(pg_stat_user_tables_seq_scan{relname=~\"napi_ips_.*\"}[5m]))

• Tuples fetched by sequential scans per second by relname

rate(pg_stat_user_tables_seq_tup_read{relname!~\"napi_ips_.*\"}[5m])
sum(rate(pg_stat_user_tables_seq_tup_read{relname=~\"napi_ips_.*\"}[5m]))

• INSERTs per second by relname

rate(pg_stat_user_tables_n_tup_ins{relname!~\"napi_ips_.*\"}[5m])
sum(rate(pg_stat_user_tables_n_tup_ins{relname=~\"napi_ips_.*\"}[5m]))

• UPDATEs per second by relname

rate(pg_stat_user_tables_n_tup_upd{relname!~\"napi_ips_.*\"}[5m])
sum(rate(pg_stat_user_tables_n_tup_upd{relname=~\"napi_ips_.*\"}[5m]))

• DELETEs per second by relname

rate(pg_stat_user_tables_n_tup_del{relname!~\"napi_ips_.*\"}[5m])
sum(rate(pg_stat_user_tables_n_tup_del{relname=~\"napi_ips_.*\"}[5m]))

Dashboard Moray

• Panel p99 request latency by instance

histogram_quantile(0.99, sum(rate(fast_request_time_ms[5m])) by (le,instance))

• Panel p99 request latency by RPC

histogram_quantile(0.99, sum(rate(fast_request_time_ms[5m])) by (le,rpcMethod))

• Panel Moray nConnectionsActive

plugin_moray_nConnectionsActive{exported_alias=\"\",alias=~\"moray.\"}

• Panel Moray nRequestsCompleted

rate(plugin_moray_nRequestsCompleted{alias=~\"moray.\",exported_alias=\"\"}[5m])

• Panel Moray nRequestsFailed

rate(plugin_moray_nRequestsFailed{exported_alias=\"\",alias=~\"moray.\"}[5m])

Dashboard NAPI

• Panel NAPI moray latency

histogram_quantile(0.90, sum(rate(fast_client_request_time_ms{service=\"napi\"}[5m])) by (le))
histogram_quantile(0.95, sum(rate(fast_client_request_time_ms{service=\"napi\"}[5m])) by (le))
histogram_quantile(0.99, sum(rate(fast_client_request_time_ms{service=\"napi\"}[5m])) by (le))

Dashboard SAPI

• Panel SAPI moray latency

histogram_quantile(0.95, sum(rate(fast_client_request_time_ms{service=\"sapi\"}[5m])) by (le))

Dashboard Triton Service Routes

• Panels cloudapi - get
• Requests

sum(rate(http_requests_completed{service=~\"[[triton_service]]\",route=\"[[route]]\",status_code!~\"5.+\"}[5m]))
sum(rate(http_requests_completed{service=~\"[[triton_service]]\",route=\"[[route]]\",status_code=~\"5.+\"}[5m]))

• Latency

(sum(rate(http_request_duration_seconds_sum{service=\"[[triton_service]]\",route=\"[[route]]\"}[5m])) / sum(rate(http_request_duration_seconds_count{service=\"[[triton_service]]\",route=\"[[route]]\"}[5m])))
histogram_quantile(0.50, sum(rate(http_request_duration_seconds{service=\"[[triton_service]]\",route=\"[[route]]\"}[1m])) by (le))
histogram_quantile(0.95, sum(rate(http_request_duration_seconds{service=\"[[triton_service]]\",route=\"[[route]]\"}[1m])) by (le))

I'd be happy to help debugging why this happens.