Skip to content

[BUGFIX] PromQL: Noinline kahanSumInc, to reduce numerical errors #16895

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 28, 2025
Merged

[BUGFIX] PromQL: Noinline kahanSumInc, to reduce numerical errors #16895

merged 1 commit into from
Jul 28, 2025

Conversation

bboreham
Copy link
Member

@bboreham bboreham commented Jul 19, 2025 •
edited
Loading

Observed on go1.24.4 darwin/arm64, the compensation variable is not correctly calculated.

Note you may have to run the tests several times to see it, to get the right ordering of series.

Since the problem went away when I tried to run under the debugger (which disables optimisations), and went away when I added sufficient printf statements (which discourages inlining), I hit upon the idea of disabling inlining for this one function.

See #16714 (comment)

It's going to slow things down a bit, but turns out IsInf is pretty slow already.

Fixes #16714.

Possibly related: https://en.wikipedia.org/wiki/Kahan_summation_algorithm#Possible_invalidation_by_compiler_optimization

@bboreham
[BUGFIX] PromQL: Noinline kahanSum, to avoid compiler optimisations
358499f 
Observed on go1.24.4 darwin/arm64, the compensation variable is not correctly calculated.

Note you may have to run the tests several times to see it, to get the right ordering of series.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
@bboreham bboreham requested a review from roidelapluie as a code owner July 19, 2025 14:26
@bboreham bboreham changed the title [BUGFIX] PromQL: Noinline kahanSumInc, to avoid compiler optimisations [BUGFIX] PromQL: Noinline kahanSumInc, to reduce numerical errors Jul 20, 2025
@darshanime
Copy link
Contributor

It's going to slow things down a bit

should we measure how much slower? if the impact is too large, maybe we can disable the potential inline only for */arm64?

@bboreham
Copy link
Member Author

Doesn't show up in the benchmarks. I added sum_over_time(a_X[1d]), where the inner loop calls Kahan.

goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/promql
cpu: Intel(R) Core(TM) i7-14700K
                                                                                                          │ 3602785a8.txt │           358499f86.txt            │
                                                                                                          │    sec/op     │    sec/op     vs base              │
RangeQuery/expr=a_one,steps=1-28                                                                              4.949µ ± 8%   5.331µ ±  1%       ~ (p=0.093 n=6)
RangeQuery/expr=a_one,steps=1000-28                                                                           28.88µ ± 1%   28.47µ ±  1%  -1.43% (p=0.002 n=6)
RangeQuery/expr=a_hundred,steps=1-28                                                                          139.5µ ± 1%   140.1µ ±  2%       ~ (p=0.818 n=6)
RangeQuery/expr=a_hundred,steps=1000-28                                                                       2.449m ± 1%   2.455m ±  0%       ~ (p=0.699 n=6)
RangeQuery/expr=rate(a_one[1m]),steps=1-28                                                                    7.819µ ± 1%   7.848µ ±  2%       ~ (p=0.082 n=6)
RangeQuery/expr=rate(a_one[1m]),steps=1000-28                                                                 71.14µ ± 0%   71.20µ ±  1%       ~ (p=0.818 n=6)
RangeQuery/expr=rate(a_hundred[1m]),steps=1-28                                                                168.4µ ± 1%   170.4µ ±  1%  +1.19% (p=0.015 n=6)
RangeQuery/expr=rate(a_hundred[1m]),steps=1000-28                                                             6.990m ± 1%   6.971m ±  1%       ~ (p=1.000 n=6)
RangeQuery/expr=rate(a_one[1m]),steps=10000-28                                                                711.3µ ± 2%   716.4µ ±  1%       ~ (p=0.240 n=6)
RangeQuery/expr=rate(a_hundred[1m]),steps=10000-28                                                            76.10m ± 1%   76.14m ±  2%       ~ (p=0.699 n=6)
RangeQuery/expr=rate(sparse[1m]),steps=10000-28                                                               4.194m ± 0%   4.204m ±  2%       ~ (p=0.310 n=6)
RangeQuery/expr=double_exponential_smoothing(a_one[1d],_0.3,_0.3),steps=1-28                                  359.4µ ± 1%   359.9µ ±  2%       ~ (p=0.589 n=6)
RangeQuery/expr=double_exponential_smoothing(a_one[1d],_0.3,_0.3),steps=1000-28                               32.82m ± 0%   32.82m ±  0%       ~ (p=0.818 n=6)
RangeQuery/expr=double_exponential_smoothing(a_hundred[1d],_0.3,_0.3),steps=1-28                              25.47m ± 1%   25.51m ±  0%       ~ (p=0.937 n=6)
RangeQuery/expr=double_exponential_smoothing(a_hundred[1d],_0.3,_0.3),steps=1000-28                            3.245 ± 0%    3.246 ±  0%       ~ (p=0.394 n=6)
RangeQuery/expr=changes(a_one[1d]),steps=1-28                                                                 328.0µ ± 2%   329.1µ ±  1%       ~ (p=0.818 n=6)
RangeQuery/expr=changes(a_one[1d]),steps=1000-28                                                              40.74m ± 0%   40.76m ±  0%       ~ (p=1.000 n=6)
RangeQuery/expr=changes(a_hundred[1d]),steps=1-28                                                             27.11m ± 1%   26.87m ±  1%  -0.89% (p=0.026 n=6)
RangeQuery/expr=changes(a_hundred[1d]),steps=1000-28                                                           3.577 ± 0%    3.575 ±  0%       ~ (p=0.485 n=6)
RangeQuery/expr=rate(a_one[1d]),steps=1-28                                                                    256.5µ ± 1%   256.3µ ±  1%       ~ (p=0.937 n=6)
RangeQuery/expr=rate(a_one[1d]),steps=1000-28                                                                 6.150m ± 0%   6.154m ±  1%       ~ (p=0.699 n=6)
RangeQuery/expr=rate(a_hundred[1d]),steps=1-28                                                                20.14m ± 0%   20.08m ±  0%       ~ (p=0.132 n=6)
RangeQuery/expr=rate(a_hundred[1d]),steps=1000-28                                                             609.1m ± 1%   605.5m ±  1%       ~ (p=0.394 n=6)
RangeQuery/expr=absent_over_time(a_one[1d]),steps=1-28                                                        251.5µ ± 2%   251.9µ ±  2%       ~ (p=0.699 n=6)
RangeQuery/expr=absent_over_time(a_one[1d]),steps=1000-28                                                     3.589m ± 1%   3.579m ±  0%       ~ (p=0.394 n=6)
RangeQuery/expr=absent_over_time(a_hundred[1d]),steps=1-28                                                    19.42m ± 1%   19.48m ±  1%       ~ (p=0.394 n=6)
RangeQuery/expr=absent_over_time(a_hundred[1d]),steps=1000-28                                                 350.7m ± 0%   349.5m ±  0%  -0.34% (p=0.004 n=6)
RangeQuery/expr=-a_one,steps=1-28                                                                             6.267µ ± 1%   6.275µ ±  1%       ~ (p=1.000 n=6)
RangeQuery/expr=-a_one,steps=1000-28                                                                          30.37µ ± 2%   30.34µ ±  1%       ~ (p=0.485 n=6)
RangeQuery/expr=-a_hundred,steps=1-28                                                                         142.1µ ± 1%   142.2µ ±  1%       ~ (p=0.937 n=6)
RangeQuery/expr=-a_hundred,steps=1000-28                                                                      2.525m ± 1%   2.513m ±  0%       ~ (p=0.310 n=6)
RangeQuery/expr=a_one_-_b_one,steps=1-28                                                                      10.90µ ± 1%   10.88µ ±  1%       ~ (p=1.000 n=6)
RangeQuery/expr=a_one_-_b_one,steps=1000-28                                                                   227.9µ ± 3%   228.7µ ±  0%       ~ (p=0.310 n=6)
RangeQuery/expr=a_hundred_-_b_hundred,steps=1-28                                                              341.4µ ± 3%   346.6µ ±  1%       ~ (p=0.093 n=6)
RangeQuery/expr=a_hundred_-_b_hundred,steps=1000-28                                                           16.23m ± 3%   16.62m ±  1%  +2.39% (p=0.041 n=6)
RangeQuery/expr=a_one_-_b_one,steps=10000-28                                                                  2.266m ± 1%   2.285m ±  1%  +0.84% (p=0.004 n=6)
RangeQuery/expr=a_hundred_-_b_hundred,steps=10000-28                                                          173.0m ± 2%   175.3m ±  1%       ~ (p=0.065 n=6)
RangeQuery/expr=a_one_and_b_one{l=~'.*[0-4]$'},steps=1-28                                                     16.03µ ± 1%   16.08µ ±  1%       ~ (p=0.329 n=6)
RangeQuery/expr=a_one_and_b_one{l=~'.*[0-4]$'},steps=1000-28                                                  89.66µ ± 1%   90.70µ ±  1%  +1.15% (p=0.026 n=6)
RangeQuery/expr=a_hundred_and_b_hundred{l=~'.*[0-4]$'},steps=1-28                                             245.7µ ± 2%   251.5µ ±  1%  +2.35% (p=0.002 n=6)
RangeQuery/expr=a_hundred_and_b_hundred{l=~'.*[0-4]$'},steps=1000-28                                          8.708m ± 4%   8.838m ±  2%       ~ (p=0.310 n=6)
RangeQuery/expr=a_one_or_b_one{l=~'.*[0-4]$'},steps=1-28                                                      16.16µ ± 1%   16.25µ ±  1%       ~ (p=0.132 n=6)
RangeQuery/expr=a_one_or_b_one{l=~'.*[0-4]$'},steps=1000-28                                                   111.4µ ± 2%   113.3µ ±  1%  +1.77% (p=0.015 n=6)
RangeQuery/expr=a_hundred_or_b_hundred{l=~'.*[0-4]$'},steps=1-28                                              260.1µ ± 2%   266.8µ ±  1%  +2.57% (p=0.004 n=6)
RangeQuery/expr=a_hundred_or_b_hundred{l=~'.*[0-4]$'},steps=1000-28                                           10.86m ± 1%   10.82m ±  2%       ~ (p=0.937 n=6)
RangeQuery/expr=a_one_unless_b_one{l=~'.*[0-4]$'},steps=1-28                                                  16.21µ ± 4%   16.23µ ±  0%       ~ (p=0.699 n=6)
RangeQuery/expr=a_one_unless_b_one{l=~'.*[0-4]$'},steps=1000-28                                               110.9µ ± 2%   111.8µ ±  1%       ~ (p=0.132 n=6)
RangeQuery/expr=a_hundred_unless_b_hundred{l=~'.*[0-4]$'},steps=1-28                                          246.3µ ± 1%   248.9µ ±  1%  +1.02% (p=0.015 n=6)
RangeQuery/expr=a_hundred_unless_b_hundred{l=~'.*[0-4]$'},steps=1000-28                                       8.710m ± 4%   8.782m ±  1%       ~ (p=0.818 n=6)
RangeQuery/expr=a_one_and_b_one{l='notfound'},steps=1-28                                                      8.986µ ± 1%   9.034µ ±  2%       ~ (p=0.093 n=6)
RangeQuery/expr=a_one_and_b_one{l='notfound'},steps=1000-28                                                   82.55µ ± 1%   82.69µ ±  1%       ~ (p=0.240 n=6)
RangeQuery/expr=a_hundred_and_b_hundred{l='notfound'},steps=1-28                                              146.6µ ± 1%   145.5µ ±  2%       ~ (p=0.485 n=6)
RangeQuery/expr=a_hundred_and_b_hundred{l='notfound'},steps=1000-28                                           3.306m ± 3%   3.199m ±  2%  -3.24% (p=0.009 n=6)
RangeQuery/expr=abs(a_one),steps=1-28                                                                         7.152µ ± 2%   7.100µ ±  1%       ~ (p=0.240 n=6)
RangeQuery/expr=abs(a_one),steps=1000-28                                                                      88.58µ ± 1%   88.84µ ±  1%       ~ (p=0.310 n=6)
RangeQuery/expr=abs(a_hundred),steps=1-28                                                                     162.5µ ± 1%   163.0µ ±  1%       ~ (p=0.240 n=6)
RangeQuery/expr=abs(a_hundred),steps=1000-28                                                                  6.005m ± 3%   6.019m ±  2%       ~ (p=0.937 n=6)
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=1-28                                      11.68µ ± 3%   11.45µ ±  3%  -1.92% (p=0.026 n=6)
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=1000-28                                   35.22µ ± 1%   35.20µ ±  0%       ~ (p=0.418 n=6)
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=1-28                                  167.1µ ± 2%   167.7µ ±  2%       ~ (p=0.310 n=6)
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=1000-28                               2.475m ± 1%   2.485m ±  1%       ~ (p=0.240 n=6)
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=1-28                                             9.811µ ± 1%   9.833µ ±  1%       ~ (p=0.848 n=6)
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=1000-28                                          33.70µ ± 1%   33.66µ ±  1%       ~ (p=0.937 n=6)
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=1-28                                         156.7µ ± 1%   157.6µ ±  1%       ~ (p=0.485 n=6)
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=1000-28                                      2.460m ± 1%   2.464m ±  1%       ~ (p=0.394 n=6)
RangeQuery/expr=sum(a_one),steps=1-28                                                                         6.837µ ± 1%   6.803µ ±  0%       ~ (p=0.087 n=6)
RangeQuery/expr=sum(a_one),steps=1000-28                                                                      58.83µ ± 1%   58.86µ ±  1%       ~ (p=0.818 n=6)
RangeQuery/expr=sum(a_hundred),steps=1-28                                                                     136.1µ ± 2%   137.1µ ±  1%       ~ (p=0.180 n=6)
RangeQuery/expr=sum(a_hundred),steps=1000-28                                                                  2.818m ± 2%   2.840m ±  1%       ~ (p=0.065 n=6)
RangeQuery/expr=avg(a_one),steps=1-28                                                                         6.785µ ± 8%   6.792µ ±  2%       ~ (p=0.589 n=6)
RangeQuery/expr=avg(a_one),steps=1000-28                                                                      59.68µ ± 5%   59.25µ ±  0%       ~ (p=0.310 n=6)
RangeQuery/expr=avg(a_hundred),steps=1-28                                                                     137.4µ ± 0%   136.5µ ±  1%  -0.70% (p=0.004 n=6)
RangeQuery/expr=avg(a_hundred),steps=1000-28                                                                  2.907m ± 1%   2.880m ±  1%  -0.93% (p=0.041 n=6)
RangeQuery/expr=sum_without_(l)(h_one),steps=1-28                                                             23.62µ ± 1%   23.20µ ±  2%  -1.79% (p=0.026 n=6)
RangeQuery/expr=sum_without_(l)(h_one),steps=1000-28                                                          418.1µ ± 1%   412.3µ ±  1%  -1.39% (p=0.002 n=6)
RangeQuery/expr=sum_without_(l)(h_hundred),steps=1-28                                                         1.569m ± 1%   1.555m ±  1%  -0.88% (p=0.009 n=6)
RangeQuery/expr=sum_without_(l)(h_hundred),steps=1000-28                                                      37.23m ± 1%   36.18m ±  2%  -2.81% (p=0.002 n=6)
RangeQuery/expr=sum_without_(le)(h_one),steps=1-28                                                            21.02µ ± 1%   20.97µ ±  1%       ~ (p=0.818 n=6)
RangeQuery/expr=sum_without_(le)(h_one),steps=1000-28                                                         334.0µ ± 0%   332.0µ ±  1%       ~ (p=0.394 n=6)
RangeQuery/expr=sum_without_(le)(h_hundred),steps=1-28                                                        1.601m ± 1%   1.580m ±  1%  -1.29% (p=0.026 n=6)
RangeQuery/expr=sum_without_(le)(h_hundred),steps=1000-28                                                     39.75m ± 1%   39.68m ± 11%       ~ (p=1.000 n=6)
RangeQuery/expr=sum_by_(l)(h_one),steps=1-28                                                                  20.94µ ± 2%   21.10µ ±  1%       ~ (p=0.310 n=6)
RangeQuery/expr=sum_by_(l)(h_one),steps=1000-28                                                               333.8µ ± 1%   334.8µ ±  1%       ~ (p=0.589 n=6)
RangeQuery/expr=sum_by_(l)(h_hundred),steps=1-28                                                              1.590m ± 1%   1.604m ±  1%  +0.89% (p=0.026 n=6)
RangeQuery/expr=sum_by_(l)(h_hundred),steps=1000-28                                                           39.44m ± 2%   39.95m ±  2%       ~ (p=0.065 n=6)
RangeQuery/expr=sum_by_(le)(h_one),steps=1-28                                                                 23.46µ ± 1%   23.48µ ±  1%       ~ (p=0.699 n=6)
RangeQuery/expr=sum_by_(le)(h_one),steps=1000-28                                                              416.9µ ± 1%   418.0µ ±  1%       ~ (p=0.485 n=6)
RangeQuery/expr=sum_by_(le)(h_hundred),steps=1-28                                                             1.575m ± 1%   1.565m ±  1%       ~ (p=0.093 n=6)
RangeQuery/expr=sum_by_(le)(h_hundred),steps=1000-28                                                          37.43m ± 2%   37.14m ±  2%       ~ (p=0.065 n=6)
RangeQuery/expr=count_values('value',_h_one),steps=100-28                                                     803.3µ ± 2%   803.9µ ±  1%       ~ (p=0.937 n=6)
RangeQuery/expr=count_values('value',_h_hundred),steps=100-28                                                 117.0m ± 3%   115.8m ±  3%       ~ (p=0.485 n=6)
RangeQuery/expr=topk(1,_a_one),steps=1-28                                                                     8.258µ ± 2%   8.285µ ±  0%       ~ (p=0.394 n=6)
RangeQuery/expr=topk(1,_a_one),steps=1000-28                                                                  162.7µ ± 2%   163.6µ ±  2%       ~ (p=0.240 n=6)
RangeQuery/expr=topk(1,_a_hundred),steps=1-28                                                                 142.0µ ± 2%   141.9µ ±  1%       ~ (p=0.818 n=6)
RangeQuery/expr=topk(1,_a_hundred),steps=1000-28                                                              3.331m ± 0%   3.356m ±  1%       ~ (p=0.240 n=6)
RangeQuery/expr=topk(5,_a_one),steps=1-28                                                                     8.316µ ± 2%   8.320µ ±  1%       ~ (p=1.000 n=6)
RangeQuery/expr=topk(5,_a_one),steps=1000-28                                                                  162.6µ ± 5%   163.9µ ±  1%       ~ (p=0.394 n=6)
RangeQuery/expr=topk(5,_a_hundred),steps=1-28                                                                 144.7µ ± 3%   145.4µ ±  1%       ~ (p=0.310 n=6)
RangeQuery/expr=topk(5,_a_hundred),steps=1000-28                                                              4.341m ± 1%   4.376m ±  1%  +0.80% (p=0.026 n=6)
RangeQuery/expr=limitk(1,_a_one),steps=1-28                                                                   8.201µ ± 1%   8.229µ ±  0%       ~ (p=0.589 n=6)
RangeQuery/expr=limitk(1,_a_one),steps=1000-28                                                                159.4µ ± 2%   160.7µ ±  1%       ~ (p=0.394 n=6)
RangeQuery/expr=limitk(1,_a_hundred),steps=1-28                                                               139.7µ ± 1%   140.8µ ±  1%       ~ (p=0.065 n=6)
RangeQuery/expr=limitk(1,_a_hundred),steps=1000-28                                                            2.721m ± 1%   2.715m ±  2%       ~ (p=0.589 n=6)
RangeQuery/expr=limitk(5,_a_one),steps=1-28                                                                   8.172µ ± 1%   8.231µ ±  1%       ~ (p=0.145 n=6)
RangeQuery/expr=limitk(5,_a_one),steps=1000-28                                                                158.7µ ± 0%   160.8µ ±  1%  +1.35% (p=0.002 n=6)
RangeQuery/expr=limitk(5,_a_hundred),steps=1-28                                                               140.7µ ± 2%   141.4µ ±  1%       ~ (p=0.065 n=6)
RangeQuery/expr=limitk(5,_a_hundred),steps=1000-28                                                            2.852m ± 3%   2.877m ±  1%       ~ (p=0.132 n=6)
RangeQuery/expr=limit_ratio(0.1,_a_one),steps=1-28                                                            8.144µ ± 5%   8.192µ ±  1%       ~ (p=0.132 n=6)
RangeQuery/expr=limit_ratio(0.1,_a_one),steps=1000-28                                                         137.8µ ± 1%   138.7µ ±  1%       ~ (p=0.093 n=6)
RangeQuery/expr=limit_ratio(0.1,_a_hundred),steps=1-28                                                        145.8µ ± 1%   147.0µ ±  2%  +0.79% (p=0.041 n=6)
RangeQuery/expr=limit_ratio(0.1,_a_hundred),steps=1000-28                                                     4.849m ± 3%   4.900m ±  2%       ~ (p=0.065 n=6)
RangeQuery/expr=limit_ratio(0.5,_a_one),steps=1-28                                                            8.098µ ± 2%   8.165µ ±  2%  +0.83% (p=0.041 n=6)
RangeQuery/expr=limit_ratio(0.5,_a_one),steps=1000-28                                                         136.5µ ± 0%   138.7µ ±  1%  +1.62% (p=0.002 n=6)
RangeQuery/expr=limit_ratio(0.5,_a_hundred),steps=1-28                                                        156.2µ ± 1%   157.8µ ±  2%  +1.04% (p=0.041 n=6)
RangeQuery/expr=limit_ratio(0.5,_a_hundred),steps=1000-28                                                     6.570m ± 3%   6.711m ±  1%  +2.15% (p=0.041 n=6)
RangeQuery/expr=limit_ratio(-0.5,_a_one),steps=1-28                                                           8.381µ ± 1%   8.445µ ±  1%  +0.77% (p=0.041 n=6)
RangeQuery/expr=limit_ratio(-0.5,_a_one),steps=1000-28                                                        175.9µ ± 1%   177.8µ ±  1%  +1.10% (p=0.002 n=6)
RangeQuery/expr=limit_ratio(-0.5,_a_hundred),steps=1-28                                                       158.3µ ± 2%   160.1µ ±  1%       ~ (p=0.132 n=6)
RangeQuery/expr=limit_ratio(-0.5,_a_hundred),steps=1000-28                                                    6.817m ± 3%   6.982m ±  1%       ~ (p=0.093 n=6)
RangeQuery/expr=rate(a_one[1m])_+_rate(b_one[1m]),steps=1-28                                                  15.20µ ± 4%   15.30µ ±  3%       ~ (p=0.937 n=6)
RangeQuery/expr=rate(a_one[1m])_+_rate(b_one[1m]),steps=1000-28                                               300.6µ ± 1%   302.0µ ±  1%       ~ (p=0.180 n=6)
RangeQuery/expr=rate(a_hundred[1m])_+_rate(b_hundred[1m]),steps=1-28                                          401.0µ ± 1%   405.5µ ±  2%       ~ (p=0.240 n=6)
RangeQuery/expr=rate(a_hundred[1m])_+_rate(b_hundred[1m]),steps=1000-28                                       24.96m ± 2%   25.14m ±  2%       ~ (p=0.240 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_one[1m])),steps=1-28                                                   9.793µ ± 1%   9.831µ ±  3%       ~ (p=0.937 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_one[1m])),steps=1000-28                                                101.5µ ± 1%   101.9µ ±  1%       ~ (p=0.132 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_hundred[1m])),steps=1-28                                               169.5µ ± 2%   170.7µ ±  1%       ~ (p=1.000 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_hundred[1m])),steps=1000-28                                            7.341m ± 2%   7.328m ±  1%       ~ (p=0.589 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_one[1m]))_/_sum_without_(l)(rate(b_one[1m])),steps=1-28                19.53µ ± 2%   19.31µ ±  3%       ~ (p=0.485 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_one[1m]))_/_sum_without_(l)(rate(b_one[1m])),steps=1000-28             362.4µ ± 0%   364.8µ ±  1%  +0.67% (p=0.009 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_hundred[1m]))_/_sum_without_(l)(rate(b_hundred[1m])),steps=1-28        346.5µ ± 2%   349.4µ ±  1%       ~ (p=0.240 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_hundred[1m]))_/_sum_without_(l)(rate(b_hundred[1m])),steps=1000-28     14.90m ± 1%   14.96m ±  2%       ~ (p=1.000 n=6)
RangeQuery/expr=histogram_quantile(0.9,_rate(h_one[5m])),steps=1-28                                           33.81µ ± 2%   33.83µ ±  2%       ~ (p=1.000 n=6)
RangeQuery/expr=histogram_quantile(0.9,_rate(h_one[5m])),steps=1000-28                                        1.444m ± 0%   1.456m ±  1%  +0.86% (p=0.002 n=6)
RangeQuery/expr=histogram_quantile(0.9,_rate(h_hundred[5m])),steps=1-28                                       2.496m ± 1%   2.510m ±  1%       ~ (p=0.394 n=6)
RangeQuery/expr=histogram_quantile(0.9,_rate(h_hundred[5m])),steps=1000-28                                    145.1m ± 0%   146.2m ±  3%  +0.70% (p=0.015 n=6)
RangeQuery/expr=a_one_+_on(l)_group_right_a_one,steps=1-28                                                    11.94µ ± 3%   11.98µ ±  2%       ~ (p=0.394 n=6)
RangeQuery/expr=a_one_+_on(l)_group_right_a_one,steps=1000-28                                                 285.0µ ± 1%   287.1µ ±  1%       ~ (p=0.065 n=6)
RangeQuery/expr=a_hundred_+_on(l)_group_right_a_one,steps=1-28                                                155.3µ ± 1%   155.1µ ±  1%       ~ (p=0.818 n=6)
RangeQuery/expr=a_hundred_+_on(l)_group_right_a_one,steps=1000-28                                             4.659m ± 1%   4.679m ±  1%       ~ (p=0.589 n=6)
RangeQuery/expr=count({__name__!=""}),steps=1-28                                                              2.046m ± 6%   2.028m ±  3%       ~ (p=0.589 n=6)
RangeQuery/expr=count({__name__!="",l=""}),steps=1-28                                                         73.34µ ± 1%   73.43µ ±  1%       ~ (p=1.000 n=6)
RangeQuery/expr=timestamp(a_one),steps=1-28                                                                   6.737µ ± 0%   6.795µ ±  1%       ~ (p=0.143 n=6)
RangeQuery/expr=timestamp(a_one),steps=1000-28                                                                109.2µ ± 2%   108.7µ ±  1%       ~ (p=0.240 n=6)
RangeQuery/expr=timestamp(a_hundred),steps=1-28                                                               167.5µ ± 1%   167.0µ ±  3%       ~ (p=0.485 n=6)
RangeQuery/expr=timestamp(a_hundred),steps=1000-28                                                            6.050m ± 1%   6.039m ±  2%       ~ (p=0.818 n=6)
RangeQuery/expr=sum_over_time(a_one[1d]),steps=1-28                                                           272.0µ ± 2%   278.5µ ±  2%  +2.40% (p=0.026 n=6)
RangeQuery/expr=sum_over_time(a_one[1d]),steps=1000-28                                                        14.17m ± 1%   14.14m ±  1%       ~ (p=0.310 n=6)
RangeQuery/expr=sum_over_time(a_hundred[1d]),steps=1-28                                                       21.67m ± 1%   21.60m ±  0%       ~ (p=0.818 n=6)
RangeQuery/expr=sum_over_time(a_hundred[1d]),steps=1000-28                                                     1.399 ± 1%    1.402 ±  0%       ~ (p=0.132 n=6)
geomean                                                                                                       503.7µ        505.2µ        +0.31%

                                                                                                          │ 3602785a8.txt  │            358499f86.txt             │
                                                                                                          │      B/op      │      B/op       vs base              │
RangeQuery/expr=a_one,steps=1-28                                                                            5.672Ki ±   0%   5.673Ki ±   0%       ~ (p=0.232 n=6)
RangeQuery/expr=a_one,steps=1000-28                                                                         9.658Ki ±   0%   9.660Ki ±   0%       ~ (p=0.093 n=6)
RangeQuery/expr=a_hundred,steps=1-28                                                                        44.35Ki ±   0%   44.35Ki ±   0%       ~ (p=0.565 n=6)
RangeQuery/expr=a_hundred,steps=1000-28                                                                     312.2Ki ±   0%   312.1Ki ±   0%       ~ (p=0.818 n=6)
RangeQuery/expr=rate(a_one[1m]),steps=1-28                                                                  8.466Ki ±   0%   8.466Ki ±   0%       ~ (p=0.636 n=6)
RangeQuery/expr=rate(a_one[1m]),steps=1000-28                                                               12.17Ki ±   0%   12.17Ki ±   0%       ~ (p=0.264 n=6)
RangeQuery/expr=rate(a_hundred[1m]),steps=1-28                                                              69.58Ki ±   0%   69.58Ki ±   0%       ~ (p=0.970 n=6)
RangeQuery/expr=rate(a_hundred[1m]),steps=1000-28                                                           310.4Ki ±   0%   310.3Ki ±   0%       ~ (p=0.056 n=6)
RangeQuery/expr=rate(a_one[1m]),steps=10000-28                                                              53.72Ki ±   1%   53.72Ki ±   1%       ~ (p=0.937 n=6)
RangeQuery/expr=rate(a_hundred[1m]),steps=10000-28                                                          2.582Mi ±   1%   2.582Mi ±   1%       ~ (p=0.937 n=6)
RangeQuery/expr=rate(sparse[1m]),steps=10000-28                                                             41.46Ki ±   7%   41.41Ki ±   1%       ~ (p=0.180 n=6)
RangeQuery/expr=double_exponential_smoothing(a_one[1d],_0.3,_0.3),steps=1-28                                1.019Mi ±   0%   1.020Mi ±   0%       ~ (p=0.589 n=6)
RangeQuery/expr=double_exponential_smoothing(a_one[1d],_0.3,_0.3),steps=1000-28                             1.045Mi ±   0%   1.045Mi ±   0%       ~ (p=0.732 n=6)
RangeQuery/expr=double_exponential_smoothing(a_hundred[1d],_0.3,_0.3),steps=1-28                            3.122Mi ±   0%   3.122Mi ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=double_exponential_smoothing(a_hundred[1d],_0.3,_0.3),steps=1000-28                         3.375Mi ± 143%   3.379Mi ± 142%       ~ (p=0.738 n=6)
RangeQuery/expr=changes(a_one[1d]),steps=1-28                                                               560.9Ki ±   0%   560.3Ki ±   0%       ~ (p=0.132 n=6)
RangeQuery/expr=changes(a_one[1d]),steps=1000-28                                                            570.3Ki ±   3%   570.3Ki ±   2%       ~ (p=0.662 n=6)
RangeQuery/expr=changes(a_hundred[1d]),steps=1-28                                                           2.663Mi ±   1%   2.657Mi ±   0%       ~ (p=0.065 n=6)
RangeQuery/expr=changes(a_hundred[1d]),steps=1000-28                                                        3.342Mi ± 130%   3.332Mi ± 131%       ~ (p=0.818 n=6)
RangeQuery/expr=rate(a_one[1d]),steps=1-28                                                                  562.3Ki ±   0%   562.9Ki ±   0%       ~ (p=0.699 n=6)
RangeQuery/expr=rate(a_one[1d]),steps=1000-28                                                               563.4Ki ±   0%   564.7Ki ±   1%       ~ (p=0.589 n=6)
RangeQuery/expr=rate(a_hundred[1d]),steps=1-28                                                              2.674Mi ±   0%   2.664Mi ±   0%       ~ (p=0.093 n=6)
RangeQuery/expr=rate(a_hundred[1d]),steps=1000-28                                                           3.125Mi ±   7%   3.126Mi ±   7%       ~ (p=0.461 n=6)
RangeQuery/expr=absent_over_time(a_one[1d]),steps=1-28                                                      566.1Ki ±   0%   565.8Ki ±   0%       ~ (p=0.485 n=6)
RangeQuery/expr=absent_over_time(a_one[1d]),steps=1000-28                                                   580.9Ki ±   1%   579.4Ki ±   1%       ~ (p=0.699 n=6)
RangeQuery/expr=absent_over_time(a_hundred[1d]),steps=1-28                                                  2.647Mi ±   0%   2.655Mi ±   0%       ~ (p=0.310 n=6)
RangeQuery/expr=absent_over_time(a_hundred[1d]),steps=1000-28                                               4.596Mi ±   3%   4.595Mi ±   4%       ~ (p=0.450 n=6)
RangeQuery/expr=-a_one,steps=1-28                                                                           6.360Ki ±   0%   6.360Ki ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=-a_one,steps=1000-28                                                                        10.35Ki ±   0%   10.35Ki ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=-a_hundred,steps=1-28                                                                       47.32Ki ±   0%   47.32Ki ±   0%       ~ (p=0.974 n=6)
RangeQuery/expr=-a_hundred,steps=1000-28                                                                    315.1Ki ±   0%   315.1Ki ±   0%       ~ (p=0.818 n=6)
RangeQuery/expr=a_one_-_b_one,steps=1-28                                                                    12.45Ki ±   0%   12.45Ki ±   0%       ~ (p=0.394 n=6)
RangeQuery/expr=a_one_-_b_one,steps=1000-28                                                                 67.32Ki ±   0%   67.33Ki ±   0%       ~ (p=0.675 n=6)
RangeQuery/expr=a_hundred_-_b_hundred,steps=1-28                                                            184.7Ki ±   0%   184.7Ki ±   0%       ~ (p=0.331 n=6)
RangeQuery/expr=a_hundred_-_b_hundred,steps=1000-28                                                         768.3Ki ±   0%   769.6Ki ±   0%  +0.17% (p=0.009 n=6)
RangeQuery/expr=a_one_-_b_one,steps=10000-28                                                                572.8Ki ±   0%   572.3Ki ±   0%       ~ (p=0.734 n=6)
RangeQuery/expr=a_hundred_-_b_hundred,steps=10000-28                                                        5.687Mi ±   1%   5.689Mi ±   0%       ~ (p=0.909 n=6)
RangeQuery/expr=a_one_and_b_one{l=~'.*[0-4]$'},steps=1-28                                                   21.47Ki ±   0%   21.47Ki ±   0%  +0.00% (p=0.045 n=6)
RangeQuery/expr=a_one_and_b_one{l=~'.*[0-4]$'},steps=1000-28                                                72.39Ki ±   0%   72.40Ki ±   0%       ~ (p=0.615 n=6)
RangeQuery/expr=a_hundred_and_b_hundred{l=~'.*[0-4]$'},steps=1-28                                           132.4Ki ±   0%   132.4Ki ±   0%       ~ (p=0.310 n=6)
RangeQuery/expr=a_hundred_and_b_hundred{l=~'.*[0-4]$'},steps=1000-28                                        3.634Mi ±   0%   3.634Mi ±   0%  -0.01% (p=0.041 n=6)
RangeQuery/expr=a_one_or_b_one{l=~'.*[0-4]$'},steps=1-28                                                    21.55Ki ±   0%   21.55Ki ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=a_one_or_b_one{l=~'.*[0-4]$'},steps=1000-28                                                 72.48Ki ±   0%   72.48Ki ±   0%       ~ (p=0.561 n=6)
RangeQuery/expr=a_hundred_or_b_hundred{l=~'.*[0-4]$'},steps=1-28                                            143.3Ki ±   0%   143.3Ki ±   0%       ~ (p=0.165 n=6)
RangeQuery/expr=a_hundred_or_b_hundred{l=~'.*[0-4]$'},steps=1000-28                                         6.974Mi ±   0%   6.974Mi ±   0%       ~ (p=0.589 n=6)
RangeQuery/expr=a_one_unless_b_one{l=~'.*[0-4]$'},steps=1-28                                                21.57Ki ±   0%   21.58Ki ±   0%       ~ (p=0.126 n=6)
RangeQuery/expr=a_one_unless_b_one{l=~'.*[0-4]$'},steps=1000-28                                             72.51Ki ±   0%   72.51Ki ±   0%       ~ (p=0.513 n=6)
RangeQuery/expr=a_hundred_unless_b_hundred{l=~'.*[0-4]$'},steps=1-28                                        132.4Ki ±   0%   132.4Ki ±   0%  +0.01% (p=0.041 n=6)
RangeQuery/expr=a_hundred_unless_b_hundred{l=~'.*[0-4]$'},steps=1000-28                                     3.634Mi ±   0%   3.634Mi ±   0%       ~ (p=0.394 n=6)
RangeQuery/expr=a_one_and_b_one{l='notfound'},steps=1-28                                                    10.17Ki ±   0%   10.17Ki ±   0%       ~ (p=0.636 n=6)
RangeQuery/expr=a_one_and_b_one{l='notfound'},steps=1000-28                                                 61.08Ki ±   0%   61.09Ki ±   0%       ~ (p=0.084 n=6)
RangeQuery/expr=a_hundred_and_b_hundred{l='notfound'},steps=1-28                                            82.25Ki ±   0%   82.26Ki ±   0%       ~ (p=0.734 n=6)
RangeQuery/expr=a_hundred_and_b_hundred{l='notfound'},steps=1000-28                                         396.9Ki ±   0%   397.0Ki ±   0%       ~ (p=0.180 n=6)
RangeQuery/expr=abs(a_one),steps=1-28                                                                       7.058Ki ±   0%   7.058Ki ±   0%       ~ (p=0.273 n=6)
RangeQuery/expr=abs(a_one),steps=1000-28                                                                    34.51Ki ±   0%   34.51Ki ±   0%       ~ (p=0.645 n=6)
RangeQuery/expr=abs(a_hundred),steps=1-28                                                                   83.18Ki ±   0%   83.17Ki ±   0%  -0.01% (p=0.022 n=6)
RangeQuery/expr=abs(a_hundred),steps=1000-28                                                                374.5Ki ±   0%   374.5Ki ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=1-28                                    12.59Ki ±   0%   12.59Ki ±   0%       ~ (p=0.152 n=6)
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=1000-28                                 16.59Ki ±   0%   16.59Ki ±   0%       ~ (p=0.387 n=6)
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=1-28                                61.33Ki ±   0%   61.33Ki ±   0%       ~ (p=0.617 n=6)
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=1000-28                             329.2Ki ±   0%   329.2Ki ±   0%       ~ (p=0.485 n=6)
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=1-28                                           9.810Ki ±   0%   9.810Ki ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=1000-28                                        13.80Ki ±   0%   13.81Ki ±   0%       ~ (p=0.242 n=6)
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=1-28                                       56.10Ki ±   0%   56.10Ki ±   0%       ~ (p=0.524 n=6)
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=1000-28                                    323.9Ki ±   0%   323.9Ki ±   0%       ~ (p=0.589 n=6)
RangeQuery/expr=sum(a_one),steps=1-28                                                                       7.144Ki ±   0%   7.143Ki ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=sum(a_one),steps=1000-28                                                                    11.13Ki ±   0%   11.13Ki ±   0%       ~ (p=0.799 n=6)
RangeQuery/expr=sum(a_hundred),steps=1-28                                                                   54.63Ki ±   0%   54.63Ki ±   0%       ~ (p=0.255 n=6)
RangeQuery/expr=sum(a_hundred),steps=1000-28                                                                322.4Ki ±   0%   322.4Ki ±   0%       ~ (p=0.818 n=6)
RangeQuery/expr=avg(a_one),steps=1-28                                                                       7.144Ki ±   0%   7.144Ki ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=avg(a_one),steps=1000-28                                                                    11.13Ki ±   0%   11.13Ki ±   0%       ~ (p=0.139 n=6)
RangeQuery/expr=avg(a_hundred),steps=1-28                                                                   54.63Ki ±   0%   54.63Ki ±   0%       ~ (p=0.145 n=6)
RangeQuery/expr=avg(a_hundred),steps=1000-28                                                                322.4Ki ±   0%   322.5Ki ±   0%       ~ (p=0.093 n=6)
RangeQuery/expr=sum_without_(l)(h_one),steps=1-28                                                           15.80Ki ±   0%   15.80Ki ±   0%       ~ (p=0.803 n=6)
RangeQuery/expr=sum_without_(l)(h_one),steps=1000-28                                                        47.19Ki ±   0%   47.19Ki ±   0%       ~ (p=0.937 n=6)
RangeQuery/expr=sum_without_(l)(h_hundred),steps=1-28                                                       535.0Ki ±   0%   535.4Ki ±   0%       ~ (p=0.180 n=6)
RangeQuery/expr=sum_without_(l)(h_hundred),steps=1000-28                                                    3.390Mi ±   0%   3.396Mi ±   0%       ~ (p=0.310 n=6)
RangeQuery/expr=sum_without_(le)(h_one),steps=1-28                                                          12.16Ki ±   0%   12.16Ki ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=sum_without_(le)(h_one),steps=1000-28                                                       43.54Ki ±   0%   43.55Ki ±   0%       ~ (p=0.056 n=6)
RangeQuery/expr=sum_without_(le)(h_hundred),steps=1-28                                                      567.8Ki ±   0%   567.7Ki ±   0%       ~ (p=0.818 n=6)
RangeQuery/expr=sum_without_(le)(h_hundred),steps=1000-28                                                   3.422Mi ±   0%   3.421Mi ±   0%       ~ (p=0.818 n=6)
RangeQuery/expr=sum_by_(l)(h_one),steps=1-28                                                                12.10Ki ±   0%   12.10Ki ±   0%       ~ (p=0.636 n=6)
RangeQuery/expr=sum_by_(l)(h_one),steps=1000-28                                                             43.48Ki ±   0%   43.48Ki ±   0%       ~ (p=0.911 n=6)
RangeQuery/expr=sum_by_(l)(h_hundred),steps=1-28                                                            567.7Ki ±   0%   567.7Ki ±   0%       ~ (p=0.818 n=6)
RangeQuery/expr=sum_by_(l)(h_hundred),steps=1000-28                                                         3.423Mi ±   0%   3.423Mi ±   0%       ~ (p=0.818 n=6)
RangeQuery/expr=sum_by_(le)(h_one),steps=1-28                                                               15.78Ki ±   0%   15.78Ki ±   0%       ~ (p=0.071 n=6)
RangeQuery/expr=sum_by_(le)(h_one),steps=1000-28                                                            47.16Ki ±   0%   47.18Ki ±   0%  +0.04% (p=0.017 n=6)
RangeQuery/expr=sum_by_(le)(h_hundred),steps=1-28                                                           535.1Ki ±   0%   535.1Ki ±   0%       ~ (p=0.310 n=6)
RangeQuery/expr=sum_by_(le)(h_hundred),steps=1000-28                                                        3.394Mi ±   0%   3.390Mi ±   0%       ~ (p=0.132 n=6)
RangeQuery/expr=count_values('value',_h_one),steps=100-28                                                   709.4Ki ±   0%   709.4Ki ±   0%       ~ (p=0.937 n=6)
RangeQuery/expr=count_values('value',_h_hundred),steps=100-28                                               65.95Mi ±   1%   65.80Mi ±   1%       ~ (p=0.589 n=6)
RangeQuery/expr=topk(1,_a_one),steps=1-28                                                                   8.246Ki ±   0%   8.246Ki ±   0%       ~ (p=0.273 n=6)
RangeQuery/expr=topk(1,_a_one),steps=1000-28                                                                200.3Ki ±   0%   200.3Ki ±   0%       ~ (p=0.970 n=6)
RangeQuery/expr=topk(1,_a_hundred),steps=1-28                                                               66.43Ki ±   0%   66.41Ki ±   0%       ~ (p=0.115 n=6)
RangeQuery/expr=topk(1,_a_hundred),steps=1000-28                                                            522.1Ki ±   0%   522.2Ki ±   0%  +0.02% (p=0.004 n=6)
RangeQuery/expr=topk(5,_a_one),steps=1-28                                                                   8.247Ki ±   0%   8.246Ki ±   0%  -0.02% (p=0.022 n=6)
RangeQuery/expr=topk(5,_a_one),steps=1000-28                                                                200.2Ki ±   0%   200.2Ki ±   0%       ~ (p=0.965 n=6)
RangeQuery/expr=topk(5,_a_hundred),steps=1-28                                                               67.18Ki ±   0%   67.17Ki ±   0%       ~ (p=0.063 n=6)
RangeQuery/expr=topk(5,_a_hundred),steps=1000-28                                                            749.6Ki ±   0%   749.7Ki ±   0%       ~ (p=0.180 n=6)
RangeQuery/expr=limitk(1,_a_one),steps=1-28                                                                 8.254Ki ±   0%   8.254Ki ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=limitk(1,_a_one),steps=1000-28                                                              200.3Ki ±   0%   200.3Ki ±   0%       ~ (p=0.485 n=6)
RangeQuery/expr=limitk(1,_a_hundred),steps=1-28                                                             66.43Ki ±   0%   66.43Ki ±   0%       ~ (p=0.288 n=6)
RangeQuery/expr=limitk(1,_a_hundred),steps=1000-28                                                          522.2Ki ±   0%   522.2Ki ±   0%       ~ (p=0.394 n=6)
RangeQuery/expr=limitk(5,_a_one),steps=1-28                                                                 8.254Ki ±   0%   8.254Ki ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=limitk(5,_a_one),steps=1000-28                                                              200.3Ki ±   0%   200.3Ki ±   0%       ~ (p=0.855 n=6)
RangeQuery/expr=limitk(5,_a_hundred),steps=1-28                                                             67.10Ki ±   0%   67.10Ki ±   0%       ~ (p=0.675 n=6)
RangeQuery/expr=limitk(5,_a_hundred),steps=1000-28                                                          710.6Ki ±   0%   710.5Ki ±   0%       ~ (p=0.093 n=6)
RangeQuery/expr=limit_ratio(0.1,_a_one),steps=1-28                                                          8.167Ki ±   0%   8.167Ki ±   0%       ~ (p=0.364 n=6)
RangeQuery/expr=limit_ratio(0.1,_a_one),steps=1000-28                                                       153.3Ki ±   0%   153.3Ki ±   0%  -0.01% (p=0.037 n=6)
RangeQuery/expr=limit_ratio(0.1,_a_hundred),steps=1-28                                                      70.20Ki ±   0%   70.19Ki ±   0%       ~ (p=0.675 n=6)
RangeQuery/expr=limit_ratio(0.1,_a_hundred),steps=1000-28                                                   2.010Mi ±   0%   2.010Mi ±   0%       ~ (p=0.589 n=6)
RangeQuery/expr=limit_ratio(0.5,_a_one),steps=1-28                                                          8.167Ki ±   0%   8.167Ki ±   0%       ~ (p=0.545 n=6)
RangeQuery/expr=limit_ratio(0.5,_a_one),steps=1000-28                                                       153.3Ki ±   0%   153.2Ki ±   0%       ~ (p=0.115 n=6)
RangeQuery/expr=limit_ratio(0.5,_a_hundred),steps=1-28                                                      84.46Ki ±   0%   84.47Ki ±   0%       ~ (p=0.729 n=6)
RangeQuery/expr=limit_ratio(0.5,_a_hundred),steps=1000-28                                                   7.645Mi ±   0%   7.645Mi ±   0%       ~ (p=0.699 n=6)
RangeQuery/expr=limit_ratio(-0.5,_a_one),steps=1-28                                                         8.355Ki ±   0%   8.355Ki ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=limit_ratio(-0.5,_a_one),steps=1000-28                                                      200.4Ki ±   0%   200.4Ki ±   0%       ~ (p=0.732 n=6)
RangeQuery/expr=limit_ratio(-0.5,_a_hundred),steps=1-28                                                     85.12Ki ±   0%   85.12Ki ±   0%       ~ (p=0.589 n=6)
RangeQuery/expr=limit_ratio(-0.5,_a_hundred),steps=1000-28                                                  7.645Mi ±   0%   7.646Mi ±   0%       ~ (p=0.394 n=6)
RangeQuery/expr=rate(a_one[1m])_+_rate(b_one[1m]),steps=1-28                                                17.90Ki ±   0%   17.90Ki ±   0%       ~ (p=0.515 n=6)
RangeQuery/expr=rate(a_one[1m])_+_rate(b_one[1m]),steps=1000-28                                             72.22Ki ±   0%   72.22Ki ±   0%       ~ (p=0.437 n=6)
RangeQuery/expr=rate(a_hundred[1m])_+_rate(b_hundred[1m]),steps=1-28                                        230.0Ki ±   0%   229.9Ki ±   0%       ~ (p=0.485 n=6)
RangeQuery/expr=rate(a_hundred[1m])_+_rate(b_hundred[1m]),steps=1000-28                                     760.5Ki ±   0%   760.3Ki ±   0%       ~ (p=0.818 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_one[1m])),steps=1-28                                                 10.34Ki ±   0%   10.34Ki ±   0%       ~ (p=0.567 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_one[1m])),steps=1000-28                                              14.05Ki ±   0%   14.05Ki ±   0%       ~ (p=0.959 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_hundred[1m])),steps=1-28                                             80.57Ki ±   0%   80.58Ki ±   0%       ~ (p=0.193 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_hundred[1m])),steps=1000-28                                          321.0Ki ±   0%   321.0Ki ±   0%       ~ (p=0.329 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_one[1m]))_/_sum_without_(l)(rate(b_one[1m])),steps=1-28              22.80Ki ±   0%   22.80Ki ±   0%       ~ (p=0.123 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_one[1m]))_/_sum_without_(l)(rate(b_one[1m])),steps=1000-28           77.12Ki ±   0%   77.13Ki ±   0%       ~ (p=0.331 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_hundred[1m]))_/_sum_without_(l)(rate(b_hundred[1m])),steps=1-28      163.0Ki ±   0%   163.0Ki ±   0%       ~ (p=0.662 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_hundred[1m]))_/_sum_without_(l)(rate(b_hundred[1m])),steps=1000-28   691.3Ki ±   0%   691.2Ki ±   0%       ~ (p=0.485 n=6)
RangeQuery/expr=histogram_quantile(0.9,_rate(h_one[5m])),steps=1-28                                         23.30Ki ±   0%   23.30Ki ±   0%       ~ (p=0.097 n=6)
RangeQuery/expr=histogram_quantile(0.9,_rate(h_one[5m])),steps=1000-28                                      101.6Ki ±   0%   101.6Ki ±   0%       ~ (p=0.725 n=6)
RangeQuery/expr=histogram_quantile(0.9,_rate(h_hundred[5m])),steps=1-28                                     1.163Mi ±   0%   1.163Mi ±   0%       ~ (p=0.394 n=6)
RangeQuery/expr=histogram_quantile(0.9,_rate(h_hundred[5m])),steps=1000-28                                  4.097Mi ±   1%   4.092Mi ±   0%       ~ (p=0.900 n=6)
RangeQuery/expr=a_one_+_on(l)_group_right_a_one,steps=1-28                                                  13.14Ki ±   0%   13.14Ki ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=a_one_+_on(l)_group_right_a_one,steps=1000-28                                               255.7Ki ±   0%   255.7Ki ±   0%       ~ (p=0.818 n=6)
RangeQuery/expr=a_hundred_+_on(l)_group_right_a_one,steps=1-28                                              104.8Ki ±   0%   104.8Ki ±   0%       ~ (p=0.465 n=6)
RangeQuery/expr=a_hundred_+_on(l)_group_right_a_one,steps=1000-28                                           423.5Ki ±   0%   423.5Ki ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=count({__name__!=""}),steps=1-28                                                            675.3Ki ±   0%   675.7Ki ±   0%       ~ (p=0.310 n=6)
RangeQuery/expr=count({__name__!="",l=""}),steps=1-28                                                       23.27Ki ±   0%   23.27Ki ±   0%       ~ (p=0.351 n=6)
RangeQuery/expr=timestamp(a_one),steps=1-28                                                                 6.610Ki ±   0%   6.610Ki ±   0%       ~ (p=1.000 n=6)
RangeQuery/expr=timestamp(a_one),steps=1000-28                                                              96.61Ki ±   0%   96.61Ki ±   0%       ~ (p=0.615 n=6)
RangeQuery/expr=timestamp(a_hundred),steps=1-28                                                             133.9Ki ±   0%   133.9Ki ±   0%       ~ (p=0.937 n=6)
RangeQuery/expr=timestamp(a_hundred),steps=1000-28                                                          5.072Mi ±   0%   5.072Mi ±   0%       ~ (p=0.699 n=6)
RangeQuery/expr=sum_over_time(a_one[1d]),steps=1-28                                                         560.5Ki ±   0%   560.5Ki ±   0%       ~ (p=0.818 n=6)
RangeQuery/expr=sum_over_time(a_one[1d]),steps=1000-28                                                      567.6Ki ±   1%   567.6Ki ±   0%       ~ (p=0.818 n=6)
RangeQuery/expr=sum_over_time(a_hundred[1d]),steps=1-28                                                     2.659Mi ±   1%   2.658Mi ±   1%       ~ (p=1.000 n=6)
RangeQuery/expr=sum_over_time(a_hundred[1d]),steps=1000-28                                                  3.347Mi ± 144%   3.345Mi ± 145%       ~ (p=0.970 n=6)
geomean                                                                                                     151.9Ki          151.9Ki         -0.00%

                                                                                                          │ 3602785a8.txt │            358499f86.txt            │
                                                                                                          │   allocs/op   │  allocs/op   vs base                │
RangeQuery/expr=a_one,steps=1-28                                                                               112.0 ± 0%    112.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_one,steps=1000-28                                                                            144.0 ± 0%    144.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred,steps=1-28                                                                           713.0 ± 0%    713.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred,steps=1000-28                                                                       3.445k ± 0%   3.445k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=rate(a_one[1m]),steps=1-28                                                                     170.0 ± 0%    170.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=rate(a_one[1m]),steps=1000-28                                                                  199.0 ± 0%    199.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=rate(a_hundred[1m]),steps=1-28                                                                1.170k ± 0%   1.170k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=rate(a_hundred[1m]),steps=1000-28                                                             3.611k ± 0%   3.611k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=rate(a_one[1m]),steps=10000-28                                                                 491.0 ± 0%    491.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=rate(a_hundred[1m]),steps=10000-28                                                            27.55k ± 0%   27.55k ± 0%       ~ (p=0.970 n=6)
RangeQuery/expr=rate(sparse[1m]),steps=10000-28                                                                653.0 ± 0%    653.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=double_exponential_smoothing(a_one[1d],_0.3,_0.3),steps=1-28                                   494.0 ± 0%    494.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=double_exponential_smoothing(a_one[1d],_0.3,_0.3),steps=1000-28                                519.0 ± 0%    519.0 ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=double_exponential_smoothing(a_hundred[1d],_0.3,_0.3),steps=1-28                              22.30k ± 0%   22.30k ± 0%       ~ (p=0.636 n=6)
RangeQuery/expr=double_exponential_smoothing(a_hundred[1d],_0.3,_0.3),steps=1000-28                           24.67k ± 4%   24.67k ± 4%       ~ (p=0.509 n=6)
RangeQuery/expr=changes(a_one[1d]),steps=1-28                                                                  401.0 ± 0%    401.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=changes(a_one[1d]),steps=1000-28                                                               425.0 ± 0%    425.0 ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=changes(a_hundred[1d]),steps=1-28                                                             22.22k ± 0%   22.21k ± 0%       ~ (p=0.082 n=6)
RangeQuery/expr=changes(a_hundred[1d]),steps=1000-28                                                          24.58k ± 4%   24.59k ± 4%       ~ (p=0.260 n=6)
RangeQuery/expr=rate(a_one[1d]),steps=1-28                                                                     407.0 ± 0%    407.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=rate(a_one[1d]),steps=1000-28                                                                  431.0 ± 0%    431.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=rate(a_hundred[1d]),steps=1-28                                                                22.63k ± 0%   22.61k ± 0%  -0.10% (p=0.013 n=6)
RangeQuery/expr=rate(a_hundred[1d]),steps=1000-28                                                             24.97k ± 0%   24.97k ± 0%       ~ (p=0.909 n=6)
RangeQuery/expr=absent_over_time(a_one[1d]),steps=1-28                                                         402.0 ± 0%    402.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=absent_over_time(a_one[1d]),steps=1000-28                                                      426.0 ± 0%    426.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=absent_over_time(a_hundred[1d]),steps=1-28                                                    22.30k ± 0%   22.30k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=absent_over_time(a_hundred[1d]),steps=1000-28                                                 24.65k ± 0%   24.64k ± 0%       ~ (p=0.165 n=6)
RangeQuery/expr=-a_one,steps=1-28                                                                              133.0 ± 0%    133.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=-a_one,steps=1000-28                                                                           165.0 ± 0%    165.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=-a_hundred,steps=1-28                                                                          737.0 ± 0%    737.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=-a_hundred,steps=1000-28                                                                      3.469k ± 0%   3.469k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_one_-_b_one,steps=1-28                                                                       215.0 ± 0%    215.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_one_-_b_one,steps=1000-28                                                                   2.277k ± 0%   2.277k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred_-_b_hundred,steps=1-28                                                              1.839k ± 0%   1.839k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred_-_b_hundred,steps=1000-28                                                           9.303k ± 0%   9.304k ± 0%  +0.01% (p=0.015 n=6)
RangeQuery/expr=a_one_-_b_one,steps=10000-28                                                                  20.84k ± 0%   20.84k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred_-_b_hundred,steps=10000-28                                                          74.58k ± 0%   74.58k ± 0%       ~ (p=0.095 n=6)
RangeQuery/expr=a_one_and_b_one{l=~'.*[0-4]$'},steps=1-28                                                      277.0 ± 0%    277.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_one_and_b_one{l=~'.*[0-4]$'},steps=1000-28                                                  2.307k ± 0%   2.307k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred_and_b_hundred{l=~'.*[0-4]$'},steps=1-28                                             1.362k ± 0%   1.362k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred_and_b_hundred{l=~'.*[0-4]$'},steps=1000-28                                          14.50k ± 0%   14.50k ± 0%       ~ (p=0.273 n=6)
RangeQuery/expr=a_one_or_b_one{l=~'.*[0-4]$'},steps=1-28                                                       278.0 ± 0%    278.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_one_or_b_one{l=~'.*[0-4]$'},steps=1000-28                                                   2.308k ± 0%   2.308k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred_or_b_hundred{l=~'.*[0-4]$'},steps=1-28                                              1.366k ± 0%   1.366k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred_or_b_hundred{l=~'.*[0-4]$'},steps=1000-28                                           16.50k ± 0%   16.50k ± 0%       ~ (p=0.494 n=6)
RangeQuery/expr=a_one_unless_b_one{l=~'.*[0-4]$'},steps=1-28                                                   278.0 ± 0%    278.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_one_unless_b_one{l=~'.*[0-4]$'},steps=1000-28                                               2.308k ± 0%   2.308k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred_unless_b_hundred{l=~'.*[0-4]$'},steps=1-28                                          1.362k ± 0%   1.362k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred_unless_b_hundred{l=~'.*[0-4]$'},steps=1000-28                                       14.50k ± 0%   14.50k ± 0%       ~ (p=0.636 n=6)
RangeQuery/expr=a_one_and_b_one{l='notfound'},steps=1-28                                                       197.0 ± 0%    197.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_one_and_b_one{l='notfound'},steps=1000-28                                                   2.227k ± 0%   2.227k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred_and_b_hundred{l='notfound'},steps=1-28                                               901.0 ± 0%    901.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred_and_b_hundred{l='notfound'},steps=1000-28                                           5.631k ± 0%   5.632k ± 0%       ~ (p=0.567 n=6)
RangeQuery/expr=abs(a_one),steps=1-28                                                                          152.0 ± 0%    152.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=abs(a_one),steps=1000-28                                                                      1.183k ± 0%   1.183k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=abs(a_hundred),steps=1-28                                                                      756.0 ± 0%    756.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=abs(a_hundred),steps=1000-28                                                                  4.487k ± 0%   4.488k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=1-28                                       271.0 ± 0%    271.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=1000-28                                    303.0 ± 0%    303.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=1-28                                  1.263k ± 0%   1.263k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=1000-28                               3.995k ± 0%   3.995k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=1-28                                              230.0 ± 0%    230.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=1000-28                                           262.0 ± 0%    262.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=1-28                                         1.032k ± 0%   1.032k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=1000-28                                      3.764k ± 0%   3.764k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=sum(a_one),steps=1-28                                                                          145.0 ± 0%    145.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum(a_one),steps=1000-28                                                                       177.0 ± 0%    177.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum(a_hundred),steps=1-28                                                                      746.0 ± 0%    746.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum(a_hundred),steps=1000-28                                                                  3.478k ± 0%   3.478k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=avg(a_one),steps=1-28                                                                          145.0 ± 0%    145.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=avg(a_one),steps=1000-28                                                                       177.0 ± 0%    177.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=avg(a_hundred),steps=1-28                                                                      746.0 ± 0%    746.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=avg(a_hundred),steps=1000-28                                                                  3.478k ± 0%   3.478k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_without_(l)(h_one),steps=1-28                                                              235.0 ± 0%    235.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_without_(l)(h_one),steps=1000-28                                                           547.0 ± 0%    547.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_without_(l)(h_hundred),steps=1-28                                                         6.777k ± 0%   6.778k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=sum_without_(l)(h_hundred),steps=1000-28                                                      36.72k ± 0%   36.72k ± 0%       ~ (p=0.385 n=6)
RangeQuery/expr=sum_without_(le)(h_one),steps=1-28                                                             220.0 ± 0%    220.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_without_(le)(h_one),steps=1000-28                                                          532.0 ± 0%    532.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_without_(le)(h_hundred),steps=1-28                                                        6.878k ± 0%   6.878k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=sum_without_(le)(h_hundred),steps=1000-28                                                     36.82k ± 0%   36.82k ± 0%       ~ (p=0.589 n=6)
RangeQuery/expr=sum_by_(l)(h_one),steps=1-28                                                                   218.0 ± 0%    218.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_by_(l)(h_one),steps=1000-28                                                                530.0 ± 0%    530.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_by_(l)(h_hundred),steps=1-28                                                              6.876k ± 0%   6.875k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=sum_by_(l)(h_hundred),steps=1000-28                                                           36.82k ± 0%   36.82k ± 0%       ~ (p=0.729 n=6)
RangeQuery/expr=sum_by_(le)(h_one),steps=1-28                                                                  237.0 ± 0%    237.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_by_(le)(h_one),steps=1000-28                                                               549.0 ± 0%    549.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_by_(le)(h_hundred),steps=1-28                                                             6.779k ± 0%   6.779k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=sum_by_(le)(h_hundred),steps=1000-28                                                          36.73k ± 0%   36.73k ± 0%       ~ (p=0.106 n=6)
RangeQuery/expr=count_values('value',_h_one),steps=100-28                                                     6.544k ± 0%   6.544k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=count_values('value',_h_hundred),steps=100-28                                                 568.5k ± 0%   568.5k ± 0%       ~ (p=0.781 n=6)
RangeQuery/expr=topk(1,_a_one),steps=1-28                                                                      169.0 ± 0%    169.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=topk(1,_a_one),steps=1000-28                                                                  3.198k ± 0%   3.198k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=topk(1,_a_hundred),steps=1-28                                                                  773.0 ± 0%    773.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=topk(1,_a_hundred),steps=1000-28                                                              6.503k ± 0%   6.503k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=topk(5,_a_one),steps=1-28                                                                      169.0 ± 0%    169.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=topk(5,_a_one),steps=1000-28                                                                  3.198k ± 0%   3.198k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=topk(5,_a_hundred),steps=1-28                                                                  777.0 ± 0%    777.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=topk(5,_a_hundred),steps=1000-28                                                              8.505k ± 0%   8.505k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=limitk(1,_a_one),steps=1-28                                                                    169.0 ± 0%    169.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limitk(1,_a_one),steps=1000-28                                                                3.198k ± 0%   3.198k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limitk(1,_a_hundred),steps=1-28                                                                773.0 ± 0%    773.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limitk(1,_a_hundred),steps=1000-28                                                            6.503k ± 0%   6.503k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limitk(5,_a_one),steps=1-28                                                                    169.0 ± 0%    169.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limitk(5,_a_one),steps=1000-28                                                                3.198k ± 0%   3.198k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limitk(5,_a_hundred),steps=1-28                                                                773.0 ± 0%    773.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limitk(5,_a_hundred),steps=1000-28                                                            6.503k ± 0%   6.503k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limit_ratio(0.1,_a_one),steps=1-28                                                             170.0 ± 0%    170.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limit_ratio(0.1,_a_one),steps=1000-28                                                         2.200k ± 0%   2.200k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limit_ratio(0.1,_a_hundred),steps=1-28                                                         785.0 ± 0%    785.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limit_ratio(0.1,_a_hundred),steps=1000-28                                                     10.51k ± 0%   10.51k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=limit_ratio(0.5,_a_one),steps=1-28                                                             170.0 ± 0%    170.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limit_ratio(0.5,_a_one),steps=1000-28                                                         2.200k ± 0%   2.200k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limit_ratio(0.5,_a_hundred),steps=1-28                                                         789.0 ± 0%    789.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limit_ratio(0.5,_a_hundred),steps=1000-28                                                     12.53k ± 0%   12.53k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=limit_ratio(-0.5,_a_one),steps=1-28                                                            173.0 ± 0%    173.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limit_ratio(-0.5,_a_one),steps=1000-28                                                        3.202k ± 0%   3.202k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limit_ratio(-0.5,_a_hundred),steps=1-28                                                        789.0 ± 0%    789.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=limit_ratio(-0.5,_a_hundred),steps=1000-28                                                    12.53k ± 0%   12.53k ± 0%       ~ (p=0.545 n=6)
RangeQuery/expr=rate(a_one[1m])_+_rate(b_one[1m]),steps=1-28                                                   315.0 ± 0%    315.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=rate(a_one[1m])_+_rate(b_one[1m]),steps=1000-28                                               2.371k ± 0%   2.371k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=rate(a_hundred[1m])_+_rate(b_hundred[1m]),steps=1-28                                          2.745k ± 0%   2.745k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=rate(a_hundred[1m])_+_rate(b_hundred[1m]),steps=1000-28                                       9.623k ± 0%   9.623k ± 0%       ~ (p=0.364 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_one[1m])),steps=1-28                                                    208.0 ± 0%    208.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_without_(l)(rate(a_one[1m])),steps=1000-28                                                 237.0 ± 0%    237.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_without_(l)(rate(a_hundred[1m])),steps=1-28                                               1.213k ± 0%   1.213k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_without_(l)(rate(a_hundred[1m])),steps=1000-28                                            3.650k ± 0%   3.650k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=sum_without_(l)(rate(a_one[1m]))_/_sum_without_(l)(rate(b_one[1m])),steps=1-28                 398.0 ± 0%    398.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_without_(l)(rate(a_one[1m]))_/_sum_without_(l)(rate(b_one[1m])),steps=1000-28             2.454k ± 0%   2.454k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_without_(l)(rate(a_hundred[1m]))_/_sum_without_(l)(rate(b_hundred[1m])),steps=1-28        2.403k ± 0%   2.403k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_without_(l)(rate(a_hundred[1m]))_/_sum_without_(l)(rate(b_hundred[1m])),steps=1000-28     9.280k ± 0%   9.280k ± 0%       ~ (p=0.182 n=6)
RangeQuery/expr=histogram_quantile(0.9,_rate(h_one[5m])),steps=1-28                                            371.0 ± 0%    371.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=histogram_quantile(0.9,_rate(h_one[5m])),steps=1000-28                                        2.681k ± 0%   2.681k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=histogram_quantile(0.9,_rate(h_hundred[5m])),steps=1-28                                       12.29k ± 0%   12.29k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=histogram_quantile(0.9,_rate(h_hundred[5m])),steps=1000-28                                    44.23k ± 0%   44.23k ± 0%       ~ (p=0.861 n=6)
RangeQuery/expr=a_one_+_on(l)_group_right_a_one,steps=1-28                                                     235.0 ± 0%    235.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_one_+_on(l)_group_right_a_one,steps=1000-28                                                 4.295k ± 0%   4.295k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred_+_on(l)_group_right_a_one,steps=1-28                                                 936.0 ± 0%    936.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=a_hundred_+_on(l)_group_right_a_one,steps=1000-28                                             5.699k ± 0%   5.699k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=count({__name__!=""}),steps=1-28                                                              8.825k ± 0%   8.825k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=count({__name__!="",l=""}),steps=1-28                                                          247.0 ± 0%    247.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=timestamp(a_one),steps=1-28                                                                    145.0 ± 0%    145.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=timestamp(a_one),steps=1000-28                                                                3.174k ± 0%   3.174k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=timestamp(a_hundred),steps=1-28                                                               1.158k ± 0%   1.158k ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=timestamp(a_hundred),steps=1000-28                                                            6.896k ± 0%   6.896k ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=sum_over_time(a_one[1d]),steps=1-28                                                            401.0 ± 0%    401.0 ± 0%       ~ (p=1.000 n=6) ¹
RangeQuery/expr=sum_over_time(a_one[1d]),steps=1000-28                                                         426.0 ± 0%    426.0 ± 0%       ~ (p=1.000 n=6)
RangeQuery/expr=sum_over_time(a_hundred[1d]),steps=1-28                                                       22.21k ± 0%   22.22k ± 0%       ~ (p=0.734 n=6)
RangeQuery/expr=sum_over_time(a_hundred[1d]),steps=1000-28                                                    24.58k ± 4%   24.58k ± 4%       ~ (p=0.974 n=6)
geomean                                                                                                       1.621k        1.621k       -0.00%
¹ all samples are equal

roidelapluie
roidelapluie approved these changes Jul 28, 2025
View reviewed changes
Copy link
Member

@roidelapluie roidelapluie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice findings

@bboreham bboreham merged commit e9cf792 into prometheus:main Jul 28, 2025
27 checks passed
@bboreham bboreham deleted the noinline-kahan branch July 28, 2025 14:55
@beorn7 beorn7 mentioned this pull request Jul 31, 2025
Open
@beorn7
Copy link
Member

beorn7 commented Jul 31, 2025

Thanks for this discovery. From my semi-informed perspective, this seems like a plausible explanation. With the Kahan trick inlined, I could see how the compiler says at some point: "Hey, this compensation term is mathematically always zero, I can just optimize it away."

@beorn7 beorn7 mentioned this pull request Jul 31, 2025
Closed
@charleskorn charleskorn mentioned this pull request Aug 1, 2025
Merged
charleskorn added a commit to grafana/mimir that referenced this pull request Aug 1, 2025
@charleskorn
MQE: reduce numerical errors by not inlining KahanSumInc (#12268)
0a5adbe 
#### What this PR does

See prometheus/prometheus#16895 for further
explanation. We've copied the code into MQE, so we need to apply the
same change here as well.

#### Which issue(s) this PR fixes or relates to

prometheus/prometheus#16895

#### Checklist

- [n/a] Tests updated.
- [n/a] Documentation added.
- [n/a] `CHANGELOG.md` updated - the order of entries should be
`[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry
is not needed, please add the `changelog-not-needed` label to the PR.
- [n/a]
[`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md)
updated with experimental features.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@roidelapluie roidelapluie roidelapluie approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

promql: Numerical accuracy issues with mean calculation
4 participants
@bboreham @darshanime @beorn7 @roidelapluie