graphite returns no datapoints if time range includes only datapoints in carbon cache

[cout]

We have a carbon cache which under normal conditions holds about 9 hours of metrics in each queue before flushing to disk. When querying using graphite-web, no datapoints are returned if the time window is less than 9 hours. For example:

curl 'http://graphite-host:8888/render?target=carbon.agents.*.cache.size&from=-1h&until=now&format=json'
[]

The reason this happens is because WhisperReader.get_intervals() assumes that the newest datapoint is the modification time of the whisper file; it does not query the cache to see if it has any newer datapoints.

I'm using an older version of graphite-web (git master circa Oct 2015), but from what I can tell the limitation still exists in master. This is my crude workaround:

pbrannan@node03-pre:~/graphite-web-prod/webapp/graphite$ git diff readers.py
diff --git a/webapp/graphite/readers.py b/webapp/graphite/readers.py
index 537c6c9..e1060cf 100644
--- a/webapp/graphite/readers.py
+++ b/webapp/graphite/readers.py
@@ -152,8 +152,13 @@ class WhisperReader(object):
     self.real_metric_path = real_metric_path
 
   def get_intervals(self):
+    cached_datapoints = CarbonLink.query(self.real_metric_path)
+    if isinstance(cached_datapoints, dict):
+      cached_datapoints = cached_datapoints.items()
+    cached_timestamps = [ timestamp for (timestamp, value) in cached_datapoints ]
+    max_cached_timestamp = max(cached_timestamps)
     start = time.time() - whisper.info(self.fs_path)['maxRetention']
-    end = max( os.stat(self.fs_path).st_mtime, start )
+    end = max( os.stat(self.fs_path).st_mtime, start, max_cached_timestamp )
     return IntervalSet( [Interval(start, end)] )

I don't like this, because now the cache is always queried, even if the end of the interval is older than the modification time on the whisper file. A better fix would be to somehow pass the query interval into get_intervals(), but this would break the API for finders.

An alternative workaround is to adjust FIND_TOLERANCE to 24 hours (24 * 60 * 60), but afaict this is not well-documented. Given the number of unanswered questions around the web about metrics mysteriously not showing up until they are flushed, it might be worthwhile mentioning this in the documentation as a possible solution.

This issue is similar to but slightly different from other issues I've found:

• Metric queries return null data until whisper file exists #629 - related, but specifically mentions the creation of new metrics
• parsing ':' in time url parameters needs to be python 2.4 compat #602 - also related, but I believe this issue and the proposed patch are related to hashing
• Recent metric data not displayed until flushed to disk #1460 - related, but in this case queries were coming into the cache and returning successfully; in the example above, the cache is never queried at all

[deniszh]

That's strange - if that be true then cache was not working completely, right?
I saw quite big carbon caches in memory and never experience this issue. Which version do you running, @cout ?

[deniszh]

#629 is a different issue, not sure why #602 is there too - typo in issue number?
#1460 is a complete mystery to me too, looks like some cache misconfiguration.

[deniszh]

Maybe it's worth to share your config too. Thanks!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.