Correlate Synapse version under each Grafana graph #15662
Description
Spawning from #15657 (comment),
What's possible:
We've rediscovered the synapse_build_info metric (introduced in #6005) which tracks the deployed Synapse version over time. I've added a graph that simply displays this which is already a big win:
But this can also be mixed into existing time-series for easier visual correlation (following this guide):
We can also additionally add annotations which can be toggled across all panels from a single toggle at the top of the dashboard. Dashboard settings -> Annotations -> New query. Set the Title as Deployed {{version}} and use the following Prometheus query:
changes(process_start_time_seconds{instance="matrix.org",job=~"synapse"}[$bucket_size]) * on (instance, job) group_left(version) synapse_build_info{instance="matrix.org",job="synapse"}
(thanks to https://www.robustperception.io/exposing-the-software-version-to-prometheus/ for the join trick)
What do we want to do?
The question is, do we want this added to all graphs? A select few?
The annotations are toggle-able and will show across all panels which means we probably don't have to add the full thick bottom color to many graphs.
The matrix.org dashboard has some of these changes so far if you want to see how it all feels: annotations, a new "Deployed Synapse vresions over time" graph, and updated "Up" graph, changes so far if you want to see how it all feels.
Limitations
Unable to share query
Ideally, we would be able to share the query or even time-series to other graphs but it's not possible to use both our normal datasource and the -- Dashboard -- to share the query at the same time. Even when using -- Mixed --. This is a limitation of Grafana which is tracked by grafana/grafana#63866
So we would need to manually/semi-manually add this to every graph (query and series override) which can be done I just want to gauge interest before I plow ahead.
Related docs: https://grafana.com/docs/grafana/latest/datasources/
Doesn't work with every visualization
Another limitation is that I can't figure out how to add this to certain visualizations like heatmaps. There doesn't seem to be a way to have multiple visualizations in the same panel or overlay things. Luckily, most of our things are time-series based anyway so we're mostly good.
Annotations don't show on heatmaps either -> grafana/grafana#13895
As a ugly workaround, we could place a slim panel showing the versions underneath the heatmap: