Skip to content

Mimir query engine: add support for binary operations between instant vectors and scalars #9140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
Sep 5, 2024
Merged

Mimir query engine: add support for binary operations between instant vectors and scalars #9140

merged 11 commits into from
Sep 5, 2024

Conversation

charleskorn
Copy link
Contributor

@charleskorn charleskorn commented Aug 30, 2024 •
edited
Loading

What this PR does

This PR adds support in MQE for expressions like 2 * metric or metric - 4.

Performance is up to 60% faster for floats and 40% faster for native histograms compared to Prometheus' engine:

goos: darwin
goarch: arm64
pkg: github.com/grafana/mimir/pkg/streamingpromql/benchmarks
cpu: Apple M1 Pro
                                               │      Prometheus      │               Mimir                │
                                               │        sec/op        │   sec/op     vs base               │
Query/2_*_a_1,_instant_query                              146.8µ ± 2%   143.3µ ± 4%        ~ (p=0.180 n=6)
Query/2_*_a_1,_range_query_with_100_steps                 170.3µ ± 2%   149.7µ ± 3%  -12.08% (p=0.002 n=6)
Query/2_*_a_1,_range_query_with_1000_steps                378.6µ ± 4%   208.2µ ± 2%  -45.00% (p=0.002 n=6)
Query/2_*_a_100,_instant_query                            798.8µ ± 1%   776.2µ ± 1%   -2.83% (p=0.002 n=6)
Query/2_*_a_100,_range_query_with_100_steps               1.838m ± 1%   1.337m ± 1%  -27.26% (p=0.002 n=6)
Query/2_*_a_100,_range_query_with_1000_steps             10.523m ± 3%   5.841m ± 1%  -44.49% (p=0.002 n=6)
Query/2_*_a_2000,_instant_query                           10.73m ± 1%   10.43m ± 1%   -2.85% (p=0.002 n=6)
Query/2_*_a_2000,_range_query_with_100_steps              32.76m ± 1%   20.41m ± 2%  -37.68% (p=0.002 n=6)
Query/2_*_a_2000,_range_query_with_1000_steps             257.4m ± 2%   103.4m ± 1%  -59.85% (p=0.002 n=6)
Query/nh_1_/_2,_instant_query                             181.3µ ± 1%   177.7µ ± 2%   -2.02% (p=0.009 n=6)
Query/nh_1_/_2,_range_query_with_100_steps                266.7µ ± 1%   244.7µ ± 1%   -8.26% (p=0.002 n=6)
Query/nh_1_/_2,_range_query_with_1000_steps               941.0µ ± 1%   765.3µ ± 1%  -18.67% (p=0.002 n=6)
Query/nh_100_/_2,_instant_query                           4.526m ± 1%   4.489m ± 0%   -0.84% (p=0.002 n=6)
Query/nh_100_/_2,_range_query_with_100_steps             10.208m ± 1%   9.361m ± 0%   -8.30% (p=0.002 n=6)
Query/nh_100_/_2,_range_query_with_1000_steps             67.49m ± 1%   52.69m ± 3%  -21.93% (p=0.002 n=6)
Query/nh_2000_/_2,_instant_query                          83.56m ± 1%   82.66m ± 0%   -1.08% (p=0.002 n=6)
Query/nh_2000_/_2,_range_query_with_100_steps             236.5m ± 1%   172.5m ± 1%  -27.06% (p=0.002 n=6)
Query/nh_2000_/_2,_range_query_with_1000_steps             1.789 ± 3%    1.042 ± 2%  -41.73% (p=0.002 n=6)
geomean                                                   5.860m        4.533m       -22.65%

Peak memory utilisation is up to 20% lower for floats and 40% lower for native histograms compared to Prometheus' engine:

                                               │      Prometheus      │                Mimir                │
                                               │          B           │      B        vs base               │
Query/2_*_a_1,_instant_query                             72.02Mi ± 1%   71.23Mi ± 1%   -1.11% (p=0.002 n=6)
Query/2_*_a_1,_range_query_with_100_steps                70.60Mi ± 1%   71.18Mi ± 1%   +0.82% (p=0.026 n=6)
Query/2_*_a_1,_range_query_with_1000_steps               67.58Mi ± 1%   69.45Mi ± 1%   +2.76% (p=0.002 n=6)
Query/2_*_a_100,_instant_query                           65.76Mi ± 0%   65.52Mi ± 2%   -0.36% (p=0.015 n=6)
Query/2_*_a_100,_range_query_with_100_steps              66.27Mi ± 1%   65.34Mi ± 2%        ~ (p=0.071 n=6)
Query/2_*_a_100,_range_query_with_1000_steps             70.52Mi ± 2%   67.69Mi ± 2%   -4.02% (p=0.002 n=6)
Query/2_*_a_2000,_instant_query                          66.97Mi ± 2%   66.91Mi ± 2%        ~ (p=0.818 n=6)
Query/2_*_a_2000,_range_query_with_100_steps             79.73Mi ± 2%   75.52Mi ± 2%   -5.29% (p=0.002 n=6)
Query/2_*_a_2000,_range_query_with_1000_steps            169.8Mi ± 0%   133.5Mi ± 8%  -21.37% (p=0.002 n=6)
Query/nh_1_/_2,_instant_query                            78.15Mi ± 1%   78.55Mi ± 1%        ~ (p=0.699 n=6)
Query/nh_1_/_2,_range_query_with_100_steps               71.20Mi ± 1%   70.12Mi ± 1%   -1.50% (p=0.002 n=6)
Query/nh_1_/_2,_range_query_with_1000_steps              70.70Mi ± 1%   69.92Mi ± 1%        ~ (p=0.056 n=6)
Query/nh_100_/_2,_instant_query                          68.03Mi ± 1%   68.04Mi ± 1%        ~ (p=1.000 n=6)
Query/nh_100_/_2,_range_query_with_100_steps             77.12Mi ± 1%   72.78Mi ± 1%   -5.62% (p=0.002 n=6)
Query/nh_100_/_2,_range_query_with_1000_steps            170.4Mi ± 2%   119.3Mi ± 1%  -30.00% (p=0.002 n=6)
Query/nh_2000_/_2,_instant_query                         70.79Mi ± 2%   70.71Mi ± 2%        ~ (p=0.818 n=6)
Query/nh_2000_/_2,_range_query_with_100_steps            285.1Mi ± 1%   179.1Mi ± 1%  -37.17% (p=0.002 n=6)
Query/nh_2000_/_2,_range_query_with_1000_steps          1137.9Mi ± 0%   776.8Mi ± 3%  -31.74% (p=0.002 n=6)
geomean                                                  98.58Mi        90.13Mi        -8.57%

Which issue(s) this PR fixes or relates to

(none)

Checklist

  • Tests updated.
  • [n/a] Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • [n/a] about-versioning.md updated with experimental features.

@charleskorn charleskorn force-pushed the charleskorn/mqe-scalar-binops branch from 974883b to 2df2e21 Compare August 30, 2024 06:03
@charleskorn charleskorn changed the title Mimir query engine: add support for binary operations between vectors and scalars Mimir query engine: add support for binary operations between instant vectors and scalars Aug 30, 2024
@charleskorn charleskorn marked this pull request as ready for review August 30, 2024 06:03
@charleskorn charleskorn requested a review from a team as a code owner August 30, 2024 06:03
jhesketh
jhesketh reviewed Aug 30, 2024
View reviewed changes
@charleskorn charleskorn enabled auto-merge (squash) September 5, 2024 23:12
@charleskorn charleskorn merged commit dbcf9a8 into main Sep 5, 2024
29 checks passed
@charleskorn charleskorn deleted the charleskorn/mqe-scalar-binops branch September 5, 2024 23:41
@charleskorn charleskorn mentioned this pull request Dec 4, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jhesketh jhesketh jhesketh approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@charleskorn @jhesketh