Grafana shows stale status after shutdown

[wesketchum]

After shutting down and exiting nanorc, can see that the state in Grafana can show something 'old'. It would be good to have it recognize that the applications / nanorc state has been terminated.

[mroda88]

Hi @wesketchum. I updated the dashboard in v5.0.0 to restrict the information on the status only in the last 15 seconds, rather than the whole time window selected in the top-right. When no information is found in the database, the system will show

Let me know if that can be a good solution for now. Once the new interface will be in place, it will be easy to have the system to actually publish a "Shutdown" state.
We can change the word, if you prefer.

[mroda88]

This is the link to the PR DUNE-DAQ/grafana-dashboards#47
If you are happy with this, we can close the issue.

[wesketchum]

This seems reasonable to me, @mroda88. One thing I was wondering, which is maybe a question on the state machine, is if the reason we saw this before was that some status was not being sent at shutdown that was not being captured. One would hope that a reading of the 'most recent' message would get the right thing in cases where things didn't abruptly terminate/fail.

[mroda88]

So, the system does not publish an actual shutdown. It only publish initial, configured and running. Publishing a last message before shutting down would be an event-driven metric which is not possible to generate in the current system. With the new design we discussed last week, we'll be able to publish even-driven metrics: e.g. we will be able to publish an actual "Shutdown" state once the command is received. Until then we simply have to check only the recent history and if nothing is published, grafana will guess that the session is down.

[wesketchum]

OK, sounds good to me if this is the best we can do for now. Will close this then.

Thanks!