Mimir Query Engine #10067
Description
We're building a new query engine to improve the peak memory consumption of queriers.
The work has already progressed and is fairly mature. A good description of the new engine is found here:
https://github.com/grafana/mimir/blob/main/pkg/streamingpromql/README.md
This issue is to track the PRs and large number of changes involved. It will be an index instead of maintaining a list of PRs in the CHANGELOG.
Main / Unreleased
2.17.0
- Add double_exponential_smoothing MQE function #10873
- MQE: set correct lookback delta on statement #10983
- MQE: avoid using statement in
Querywhere possible #11001 - MQE: remove unneeded feature toggles #11002
- MQE: refactor function handling #11003
- MQE: initial query planner implementation #11004
- MQE: address feedback on query planner #11124
- MQE: emit query planning latency metrics #11127
- MQE: introduce
ResultTypemethod for query plannerNode#11132 - MQE: close inner operators of binary operations as early as possible #11162
- MQE: ensure aggregations release any intermediate state if they are closed before all series are read #11171
- MQE: fix issue where benchmark tool could run unexpected benchmarks #11188
- MQE: initial implementation of common subexpression elimination #11189
- MQE: don't hold reference to data in
sort/sort_descfor any longer than necessary #11190 - MQE: make
nilchecks consistent in operatorClosemethods #11204 - MQE: include query planning and materialization in activity tracker #11212
- MQE: remove flag to disable support for specific functions #11223
- MQE: ensure binary operations release any intermediate state if they are closed before all series are read #11224
- MQE: fix issue where query planner does not emit metrics or record activities #11361
- MQE: fix issue where slice is used after being returned to pool #11408
- MQE: make capitalisation of
cortex_mimir_query_engine_plan_stage_latency_secondslabel values consistent #11410 - MQE: skip decoding histogram buckets where possible #11587
- MQE: include additional information when panicking because estimated memory consumption is negative #11615
- MQE: eager loading of selectors #11624
- MQE: fix possible panic while evaluating a query containing
group_leftorgroup_right#11690 - MQE: include trace ID in panic message if memory consumption estimate becomes negative #11713
- MQE: track memory consumption of
[]promql.Bucketand[]promql.Bucketsseparately #11746 - MQE: always use query planner #11762
- MQE: fix issue where queries containing a duplicate expression and
timestamp()produce incorrect results #11830 - MQE: add tests for common subexpression elimination for expressions with multiple children, and add metric on number of selectors eliminated #11879
- MQE: adjust behaviour of binary operations with
on (__name__)to match Prometheus #11956 - MQE: fix issue where a query with a common subexpression could return incorrect results #11989
2.16.0
- MQE: add support for
group_leftandgroup_right(aka many-to-one and one-to-many matching) #10119 - MQE: Add support for histogram_stddev function #10139
- MQE: eliminate loops in
BucketedPool#10199 - MQE: don't select more data than strictly necessary #10209
- MQE: Add support for histogram_stdvar #10211
- MQE: track number of processed samples in each query #10232
- MQE: Fix panic in loading too many samples #10261
- MQE: Add support to disable functions+aggregations by name #10384
- MQE time related functions #10486
- MQE: add support for
sortandsort_desc#10487 - MQE: code cleanup #10488
- MQE: add support for
predict_linear#10503 - MQE Absent function #10523
- MQE: add support for
timestamp#10532 - MQE: address post-merge feedback on #10533 #10553
- MQE: fix issue where
group_left/group_rightcould return series with incorrect labels #10558 - MQE: Add support for label_join #10587
- MQE: add support for
topkandbottomk#10614 - MQE: add support for
stddev_over_timeandstdvar_over_time#10628 - MQE: add support for
quantile_over_time#10629 - MQE: address PR feedback on
topk/bottomk#10679 - MQE: add support for
count_values#10744 - MQE: Add quantile aggregation #10755
- MQE: Implement absent_over_time #10841
- MQE: Rename param for consistency #10885
- MQE: Use upstream quantile functions #10886
- Fix issue where a query containing a common subexpression that returned an error could also panic with
integer divide by zero#11514 - MQE: label each source of memory consumption to make it easier to identify where something is returned to the pool multiple times #11654
2.15.0
- MQE: Add support for stddev #9367
- MQE: Add tests for annotations on avg #9368
- MQE: add support for comparison operations #9398
- MQE: remove obsolete feature flags #9399
- MQE: Tidy up test case #9403
- MQE: Add helper for idx calcs #9417
- MQE: Add extra comparison op edge case tests #9418
- MQE: Add support for stdvar #9419
- MQE: Add support for increase function #9420
- MQE: fix incorrect query results or "found duplicate series for the match group" errors when binary operation has unsorted labels in
on#9482 - MQE: separate feature flag for binary comparison operations #9504
- MQE: include more information in query stats log lines #9505
- MQE: add support for
and#9507 - MQE: fix panic in
rateorincreaseover native histograms with changing schema #9518 - MQE: only reduce resolution of delta in
rateandincreaseonce #9531 - MQE: Add support for label_replace #9532
- MQE: add support for
unless#9533 - MQE: fix issue where
rateorincreasecan return multiple series with the same labels #9553 - MQE: introduce series metadata function type #9558
- MQE: don't reuse
FloatHistograminstances for successive points in instant vector selectors #9588 - MQE: add support for
or#9589 - MQE: Refactor functions over instant vectors to support scalar args and add clamp function #9639
- MQE: Add support for clamp_min and clamp_max #9641
- MQE: Fix iterating over scalar values in Clamp #9642
- MQE: Add support for round #9651
- MQE: subqueries #9664
- MQE: add support for unary addition #9681
- MQE: create slices with capacity for number of steps remaining, not all possible steps #9717
- MQE: improve performance of subqueries when the parent query is a range query with many steps #9719
- MQE: Add support for changes function #9724
- MQE: fix
orwhere both sides have series with the same labels #9874 - MQE: Add support for histogram_quantile #9929
- MQE: fix issue where subqueries could return series with no points #9998
- MQE: create slices with capacity for number of steps remaining, not all possible steps for functions over range vectors #10007
- MQE: mangle native histograms in returned slices to catch use-after-return bugs + fix issues found #10010
- MQE: Add support for histogram_avg #10046
- MQE: Ensure scalar operators close their inner operators #10047
- MQE: Add support for histogram_fraction #10048
- MQE debt: use time range to determine argument value #10050
2.14.0
- Mimir query engine: Add PromQL engine test cases from Prometheus #8422
- Mimir query engine: add support for basic trigonometric and mathematical functions #8430
- Mimir query engine: add HTTP header to force use of Prometheus' engine #8454
- Mimir query engine: update description for
-querier.enable-promql-engine-fallbackCLI flag #8455 - Streaming engine: Native histogram aggregations #8360
- MQE: Refactor aggregations #8490
- MQE: Histogram range vectors #8508
- MQE: Add support for rate on native histograms #8577
- Mimir query engine: annotations #8660
- Mimir query engine: add support for aggregations with
without, and resolve potential for hash collisions #8671 - Mimir query engine: avoid copying native histograms where possible #8677
- MQE: Refactor range functions to allow functions other than rate #8747
- Mimir query engine: fix issue where
rate()over native histograms could panic or return incorrect results #8850 - Mimir query engine: add feature toggle for binary operations #8872
- MQE: Add NH support to binary operations #8838
- Mimir query engine: sync upstream test cases #8911
- Mimir query engine: split
LimitingPool#8909 - MQE: Use kahan in sum aggregation #8923
- Mimir query engine: add native histogram custom bucket-related annotations #8924
- Mimir query engine: add support for
count_over_time,present_over_timeandlast_over_time#8925 - MQE: Enable binary operations by default #8932
- Mimir query engine: improve performance of binary operations #8933
- Mimir query engine: add support for
min_over_time,max_over_time,sum_over_timeandavg_over_time#8934 - Mimir query engine: add support for
offsetin instant and range vector selectors #8962 - Mimir query engine: pass enabled feature toggles to engine #8986
- Mimir query engine: emit annotations for binary operations between incompatible native histograms #8993
- Mimir query engine: add support for scalar constants and
pi()#8995 - MQE: Add support for more aggregation functions #9008
- Mimir query engine: add support for
vector()andscalar()#9017 - Mimir query engine: add tool to find disabled but supported test cases #9018
- Mimir query engine: correctly handle functions that produce multiple series with the same labels #9019
- Mimir query engine: fix
multiple org IDs presenterror when MQE is used to evaluate a cross-tenant query #9120 - MQE: Panic when memory isn't managed as we expect #9121
- Mimir query engine: Introduce
FunctionOverInstantVectortype #9136 - Mimir query engine: fix issue where unsupported aggregations that take a parameter do not trigger fallback #9139
- Mimir query engine: add support for binary operations between instant vectors and scalars #9140
- MQE: Fix native histogram bugs #9145
- Mimir query engine: fix panic in
minandmaxover series containing only histograms #9191 - MQE: Also apply workaround for shared span slices to range vectors #9192
- Mimir query engine: expand
TestConcurrentQueriesto include testing queries with float samples #9194 - Mimir query engine: add support for unary negation #9196
- MQE: Add support for avg aggregation #9201
- MQE: Handle min/max NaN's same as promql #9212
- MQE: Remove workaround copying spans #9225
- Mimir query engine: fix issue where
sumandavgresults over native histograms can be corrupted #9260 - MQE: Native histogram lookback should re-use existing histograms #9272
- Mimir query engine: binary operations between two scalars #9277
- Mimir query engine: enable supported upstream test cases #9278
- Mimir query engine: extract query time range values to their own type #9280
- MQE: Add tests for various mixed data and edge cases #9281
- MQE: Add support for count aggregation #9342
- MQE: Add support for group aggregation #9343
- MQE: fix
rate()over native histograms where first point in range is a counter reset #9371 - MQE: Add support for resets function #9859
- MQE: Add support for deriv function #9858
2.13.0
- First iteration of streaming PromQL engine #7693
- Streaming PromQL engine: optionally fall back to Prometheus' engine if query is not supported #7898
- Streaming PromQL engine: extract linked list of series type #7899
- Streaming PromQL engine: add support for range vector selectors as the top-level expression #8023
- Streaming PromQL engine: sort series in matrix at end #8058
- Streaming PromQL engine: binary arithmetic operations with one-to-one matching #8096
- Streaming PromQL engine: native histograms #8121
- Mimir query engine: support query cancellation and timeout #8197
- Mimir query engine: add memory consumption per query limit #8230
- Mimir query engine: finish reorganisation started in #8230 #8247
- Mimir query engine: report estimated peak memory consumption as a metric and in traces #8270
- Mimir query engine: improve performance for case where right side of binary operation has fewer series than the left side #8276
- Mimir query engine: track active queries #8277
- Mimir query engine: rename metrics and CLI flag values to match new name #8291
- Mimir query engine: report queries rejected due to hitting the memory consumption limit in the
cortex_querier_queries_rejected_totalmetric #8303 - Mimir query engine: extend binary operation tests to assert on error messages #8340
- Mimir Query Engine: Add instant-vector functions #8256
- Mimir query engine: complete renaming from #8256 #8348