collectd disk metrics are showing distorted rates

[tgrabiec]

Example:

The actual rate is around 400 and doesn't vary as much in reality as it is shown on the graph.

The rates are calculated by taking difference between consecutive samples of monotonic counters. Collectd emits samples with 10 second period, while prometheus samples them by default with 15 second period. So half of the time the period between prometheus samples will cover 1 collectd sample and half of the time it will cover 2 samples. The former will underestimate the rate and the latter will over estimate it.

To avoid this, the sampling period used by prometheus must be a multiply of collectd period. We can set the scraping interval to 10s, or modify the collectd plugin to export stats with 1s period. The latter approach is better because it gives better accuracy and works with any scrape period.

[tgrabiec]

Current value of 15s scrape period is probably chosen so that there is no distortion in Scylla metrics, which are exported by default with 3s period. So 10s period wouldn't work with that.

[tgrabiec]

Related collectd_exporter issue prometheus/collectd_exporter#24

[tzach]

Scylla 1.4 will not use collectd_exporter, so this issue might not be relevant looking forward

[amnonh]

Can we close that?