Skip to content

Conversation

kapillamba4
Copy link
Contributor

Continuation of #16562

@beorn7
Copy link
Member

beorn7 commented May 29, 2025

FTR: We'll have a detailed look here once #16562 is merged.

@beorn7
Copy link
Member

beorn7 commented Jun 18, 2025

Now this needs a rebase.

As hinted at before, I think the strict mode is in most cases too strict. Many tests don't care if there is an annotation or not. They are testing something else.

Now that eval is more tolerant, we should actually make use of it.

I think a better strategy is to do the conversion with the basic mode (in one commit) and then (in a separate commit) make those tests stricter where we are actively interested in testing the presence or absence of certain annotations.

Signed-off-by: Kapil Lamba <kapillamba4@gmail.com>
…ests

Signed-off-by: Kapil Lamba <kapillamba4@gmail.com>
@kapillamba4 kapillamba4 changed the title Convert PromQL test files to new syntax using strict migration mode Convert PromQL test files to new syntax using basic migration mode Jun 21, 2025
@kapillamba4
Copy link
Contributor Author

First commit contains changes via the script using basic mode.
The second commit introduces minor adjustments to the test files:

  • Decided to retain expect info & expect warn statements for tests involving mixed histograms and floats.
  • Explicitly added expect no_info & expect no_warn in tests producing no results.
  • Removed expect info from a test case where the metric name does not end with _total, as the test seems to focus on comparing value and not presence of annotation.
  • No changes to tests having expect fail or expect warn

The changes in the second commit were minimal, based on my understanding @beorn7
please review it when you have time, thanks !

cc @NeerajGartia21

@kapillamba4 kapillamba4 changed the title Convert PromQL test files to new syntax using basic migration mode Convert PromQL test files to new syntax via basic migration mode Jun 21, 2025
@kapillamba4 kapillamba4 changed the title Convert PromQL test files to new syntax via basic migration mode Convert PromQL tests to new syntax via basic migration mode Jun 21, 2025
Copy link
Member

@beorn7 beorn7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you.

So the one fundamental thing you have missed is that we really need to test for the absence of a warn or info if we test other expressions on the same datasets for the presence of a warn or info.

Imagine the annotation is not triggered by the expression but by some irregularity in the dataset. In this case we wouldn't notice that we also get the annotation for the supposedly annotation-free expressions, plus we would not test that the annotation is added for the right reason.

I added many small comments all over the place so that you get the idea.

{} 1

# The histogram is ignored here so there is no result but it has an info annotation now.
eval_info instant at 0m min(http_requests_histogram)
eval instant at 0m min(http_requests_histogram)
expect info

eval instant at 0m max by (group) (http_requests)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and the one below should probably also get the no_info treatment.

@@ -462,13 +489,15 @@ eval instant at 1m quantile without(point)(0.8, data)
{test="uneven samples"} 2.8

# The histogram is ignored here so the result doesn't change but it has an info annotation now.
eval_info instant at 1m quantile without(point)(0.8, {__name__=~"data(_histogram)?"})
eval instant at 1m quantile without(point)(0.8, {__name__=~"data(_histogram)?"})
expect info
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which means we should add an expect no_info to the test up in line 487 to demonstrate that normally, there is no info annotation.

@@ -681,22 +710,28 @@ load 5m
series{label="c"} {{schema:1 sum:15 count:10 buckets:[3 2 5 7 9]}}

# The histogram is ignored here so the result doesn't change but it has an info annotation now.
eval_info instant at 0m stddev(series)
eval instant at 0m stddev(series)
expect info
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And again, let's add expect no_info to the previous test in line 708 to demonstrate the difference.

@@ -534,10 +543,12 @@ eval range from 0 to 24m step 6m left_histograms == right_histograms
eval range from 0 to 24m step 6m left_histograms == bool right_histograms
{} 1 0 _ _ _

eval_info range from 0 to 24m step 6m left_histograms == right_floats_for_histograms
eval range from 0 to 24m step 6m left_histograms == right_floats_for_histograms
expect info
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So no_info on all the applicable other tests in this section.

Signed-off-by: Kapil Lamba <kapillamba4@gmail.com>
@kapillamba4
Copy link
Contributor Author

kapillamba4 commented Jul 9, 2025

Apologies for the delay here—got caught up with work and personal life. I've addressed the comments now, please take a look. Thanks for your patience!

Copy link
Member

@beorn7 beorn7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Epic work!

Let's go with the current state. Maybe we have missed a few edge cases, but those can be easily tweaked later.

@beorn7 beorn7 merged commit b7f984d into prometheus:main Jul 10, 2025
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants