Skip to content

Conversation

charleskorn
Copy link
Contributor

@charleskorn charleskorn commented Jul 9, 2024

What this PR does

This PR adds support for annotations (aka warnings) to the Mimir query engine.

It adds support for a single annotation that Prometheus' engine emits: sum() with both float and histogram samples at the same timestep for the same output series

(This PR previously also included adding an annotation for rate() over a suspected non-counter, but that will be added in a separate PR.)

The latency impact of these changes is low (<1% for our benchmarks), and the impact on peak memory utilisation is within the margin of error for our benchmarks.

This PR also adds support for comparing warnings when comparing results in query-tee.

Which issue(s) this PR fixes or relates to

(none)

Checklist

  • Tests updated.
  • [n/a] Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • [n/a] about-versioning.md updated with experimental features.

@charleskorn charleskorn force-pushed the charleskorn/annotations branch from b960936 to da1c812 Compare July 9, 2024 06:01
@charleskorn charleskorn marked this pull request as ready for review July 9, 2024 06:16
@charleskorn charleskorn requested a review from a team as a code owner July 9, 2024 06:16
Copy link
Contributor

@jhesketh jhesketh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious if we need to handle info annotations, or if only warnings come through the query engine?

for engineName, engine := range engines {
t.Run(engineName, func(t *testing.T) {
for queryType, generator := range queryTypes {
t.Run(queryType, func(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) We seem to be setting up these multi-engine, multi-query type comparison tests in a few places. I wonder if we can make some helpers?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a good idea - I'll have a think and maybe try to pull this out as a separate PR.

@charleskorn
Copy link
Contributor Author

Just curious if we need to handle info annotations, or if only warnings come through the query engine?

Both are handled as annotations in the same way, but both are emitted together in one (confusingly named) Warnings slice on the result.

The only difference between warnings and info annotations are the message: warnings start with PromQL warning and info annotations start with PromQL info. See here for an example of a warning, and here for an example of an info annotation.

Copy link
Contributor

@jhesketh jhesketh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, but if possible would love to move this behind #8577

# Conflicts:
#	CHANGELOG.md
#	tools/querytee/response_comparator.go
#	tools/querytee/response_comparator_test.go
@charleskorn
Copy link
Contributor Author

I've merged in the latest changes from main which includes handling info and warning annotations separately (prometheus/prometheus#14327).

@charleskorn charleskorn requested a review from jhesketh July 16, 2024 01:31
Copy link
Contributor

@jhesketh jhesketh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

# Conflicts:
#	CHANGELOG.md
#	pkg/streamingpromql/operators/aggregation.go
#	pkg/streamingpromql/operators/function_over_range_vector.go
@charleskorn
Copy link
Contributor Author

Putting this back to a draft - this needs to be reworked slightly for the annotation emitted by rate() when native histograms are involved

@charleskorn charleskorn marked this pull request as draft July 17, 2024 02:03
@charleskorn charleskorn marked this pull request as ready for review July 17, 2024 04:55
@charleskorn charleskorn requested a review from jhesketh July 17, 2024 04:55
@charleskorn
Copy link
Contributor Author

This is ready to go again.

Copy link
Contributor

@jhesketh jhesketh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine, but I'm not a huge fan of the haveFloats. I'm not sure that'll make sense when we make the functions generic in #8747.

If possible, can we resolve (as in come to agreement on) the open discussions on there and then decide which one we merge first and which one we re-work?

@charleskorn charleskorn force-pushed the charleskorn/annotations branch from 8fcabb2 to 8b3b88f Compare July 31, 2024 01:11
# Conflicts:
#	CHANGELOG.md
#	pkg/streamingpromql/functions.go
#	pkg/streamingpromql/operators/function_over_range_vector.go
#	tools/querytee/response_comparator_test.go
@charleskorn charleskorn enabled auto-merge (squash) July 31, 2024 02:20
@charleskorn charleskorn merged commit 1eaffce into main Jul 31, 2024
29 checks passed
@charleskorn charleskorn deleted the charleskorn/annotations branch July 31, 2024 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants