Proposal

A while ago, we introduced experimental annotations that can be emitted by the PromQL engine. An annotation can be info level and warning level, with the following intentions in mind:

• warning level annotations are for "real" issues. False positives shouldn't be possible. A query emitting a warning annotation has for sure an actual problem that the user (ideally) can and should fix.
• info level annotations might be a false positive. But they still trigger on things that deserve the user's attention. However, sometimes the circumstances they trigger on are deliberate or simply not changeable.

The two instances of info annotations are:

1. Applying a counter function (like rate) to a float metric with a name that doesn't end on _total/_sum/_count/_bucket.
2. The histogram_quantile function needs to "fix" monotonicity of buckets of a classic histogram.

Both are clearly important, but sometimes users have to deal with counters that are not named according to the naming conventions, and that's "fine" as it's just a convention. (OpenMetrics tried to get it closer to enforcement, but there are still loopholes, and the way it enforces the naming is also controversial.) Similarly, the monotonicity fix might be required because of the nature of the data coming in. Ideally, users can fix the data, but that's not always possible, and that's "fine" again.

In those cases, users are quite bothered by seeing the annotation in the UI all the time.

And that's exactly why we have the levels, to make it possible to treat the info level annotations differently from warning level annotations (e.g. allow users to opt in or out of info level annotations, or simply display them less prominently).

However, for now we have simply added all annotations to the warnings array in the API response. All info level annotation start with PromQL info and all warning level annotations start with PromQL warning, so the UI could still decide to treat them differently, but it would rely on those "magic" prefixes.

We always had in mind to add info level annotations to a newly introduced array keyed as infos in the API response. This issue intends to explore that option.

My understanding of JSON APIs so far is that adding a new key is generally considered OK and not a breaking change. Consumers that don't know that key will simply ignore it. But in principle, it is possible that some implementations will choke on the new key. (AFAIK no such issue happened when we introduced the warnings key lange ago.)

The IMHO relevant change is that consumers not aware of the new infos key will ignore it entirely, which can be seen as good or bad. The bad part is that old UIs will silently swallow those annotations, but that's at the same time the good part because UIs won't display false positives as prominently as they do now. UIs had to be updated to display info level annotations, but by doing so, they could do it in a user-friendly way (e.g. allow the user to not display them in dashboards where they are known to appear and where that is deemed "fine").

Paging @Nexucis and @juliusv as the custodians of the Prometheus UI.
Paging @Nexucis again as our local Perses expert.
@zenador you have the most background on the topic. And maybe you know whom we could ask best at Grafana Labs about the Grafana perspective on this?

Generally, it would be good to nail this down before the v3 release so we can declare annotations a stable feature with that release.