mqe: Allow usage of Mimir Query Engine in query-frontend #11417

56quarters · 2025-05-07T14:27:33Z

What this PR does

Introduce configuration to allow the Mimir Query Engine to be used instead of the Prometheus engine in the query-frontend when queries are rewritten to be shardable or subqueries are spun off.

Notable changes:

New configuration for query-frontend to allow MQE to be used with an optional fallback to the Prometheus engine (same as in queriers).
Context timeouts are now checked at least once in Selector and also every 128 series instead of only every 128 series.
Query-frontend tests that used a query engine now run with both Prometheus and MQE engines. Results are expected to be the same in most cases, excluding some minor differences like error messages.

Which issue(s) this PR fixes or relates to

N/A

Notes to reviewers

There are a lot of whitespace changes in test files: click "hide whitespace changes" when reviewing.

Checklist

Tests updated.
Documentation added.
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
about-versioning.md updated with experimental features.

pkg/frontend/querymiddleware/querysharding.go

charleskorn · 2025-05-16T08:13:29Z

pkg/mimir/modules.go

+	// Use either the Prometheus engine or Mimir Query Engine (with optional fallback to Prometheus
+	// if it has been configured) for middlewares that require executing queries using a PromQL engine.
+	var eng promql.QueryEngine
+	switch t.Cfg.Querier.QueryEngine {
+	case querier.PrometheusEngine:
+		eng = promql.NewEngine(promOpts)
+	case querier.MimirEngine:
+		streamingEngine, err := streamingpromql.NewEngine(mqeOpts, streamingpromql.NewStaticQueryLimitsProvider(0), stats.NewQueryMetrics(mqeOpts.CommonOpts.Reg), nil, util_log.Logger)
+		if err != nil {
+			return nil, fmt.Errorf("unable to create Mimir Query Engine: %w", err)
+		}

-	engineOpts, _ := engine.NewPromQLEngineOptions(t.Cfg.Querier.EngineConfig, t.ActivityTracker, util_log.Logger, promqlEngineRegisterer)
+		if t.Cfg.Querier.EnableQueryEngineFallback {
+			eng = streamingpromqlcompat.NewEngineWithFallback(streamingEngine, promql.NewEngine(promOpts), nil, util_log.Logger)
+		} else {
+			eng = streamingEngine
+		}
+	default:
+		panic(fmt.Sprintf("invalid config not caught by validation: unknown PromQL engine '%s'", t.Cfg.Querier.QueryEngine))
+	}


We have similar logic in querier.New - should we extract this to a shared method somewhere?

I was avoiding this because of the panic and the number of parameters required means this would be kinda gross. I can see how it looks.

pkg/frontend/querymiddleware/querysharding_test.go

github-actions · 2025-06-05T20:40:43Z

💻 Deploy preview deleted.

charleskorn

Overall LGTM.

Do we need to do anything to support the X-Mimir-Force-Prometheus-Engine header?

pkg/frontend/querymiddleware/querysharding_test.go

pkg/frontend/querymiddleware/spin_off_subqueries_test.go

56quarters · 2025-06-11T15:28:26Z

Do we need to do anything to support the X-Mimir-Force-Prometheus-Engine header?

I didn't know about that header so we probably do 🤦 . I'll look into it.

Introduce configuration to allow the Mimir Query Engine to be used instead of the Prometheus engine in the query-frontend when queries are rewritten to be shardable or subqueries are spun off. Notable changes: * New configuration for query-frontend to allow MQE to be used with an optional fallback to the Prometheus engine (same as in queriers). * Context timeouts are now checked at least once in `Selector` and also every 128 series instead of only every 128 series. * Query-frontend tests that used a query engine now run with both Prometheus and MQE engines. Results are expected to be the same in most cases, excluding some minor differences like error messages.

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>

pkg/mimir/modules.go

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>

#### What this PR does This PR fixes an issue where query-frontends running with MQE enabled do not emit query fallback metrics such as `cortex_mimir_query_engine_supported_queries_total` or `cortex_mimir_query_engine_unsupported_queries_total`. #### Which issue(s) this PR fixes or relates to #11417 #### Checklist - [n/a] Tests updated. - [n/a] Documentation added. - [x] `CHANGELOG.md` updated - the order of entries should be `[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry is not needed, please add the `changelog-not-needed` label to the PR. - [n/a] [`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md) updated with experimental features.

…ts log messages (#12251) #### What this PR does The query-frontend emits a `query stats` log message after each query request is completed. MQE also emits a `query stats` log message after evaluation is completed. With MQE now running in query-frontends, this is potentially confusing, so I've changed the log message from MQE to `evaluation stats` to make it unambiguous. #### Which issue(s) this PR fixes or relates to #11417 #### Checklist - [x] Tests updated. - [n/a] Documentation added. - [n/a] `CHANGELOG.md` updated - the order of entries should be `[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry is not needed, please add the `changelog-not-needed` label to the PR. - [n/a] [`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md) updated with experimental features.

Fix a copy/paste error from when MQE was added to the query-frontend. Related #11417

56quarters force-pushed the 56quarters/mqe-qfe branch from 16573ec to b5653bb Compare May 7, 2025 19:47

charleskorn reviewed May 16, 2025

View reviewed changes

charleskorn mentioned this pull request May 30, 2025

Don't try to create a Query instance just to compute stats for remote read requests #11583

Merged

56quarters mentioned this pull request Jun 5, 2025

MQE: eager loading of selectors #11624

Merged

1 task

56quarters force-pushed the 56quarters/mqe-qfe branch from 8f40489 to 62c1bf2 Compare June 5, 2025 18:51

56quarters commented Jun 5, 2025

View reviewed changes

pkg/frontend/querymiddleware/querysharding_test.go Outdated Show resolved Hide resolved

56quarters commented Jun 5, 2025

View reviewed changes

pkg/frontend/querymiddleware/querysharding_test.go Outdated Show resolved Hide resolved

56quarters force-pushed the 56quarters/mqe-qfe branch 3 times, most recently from c07f9e7 to 1e3b6cd Compare June 10, 2025 20:49

56quarters marked this pull request as ready for review June 10, 2025 21:35

56quarters requested review from tacole02 and a team as code owners June 10, 2025 21:35

charleskorn reviewed Jun 10, 2025

View reviewed changes

pkg/frontend/querymiddleware/querysharding_test.go Outdated Show resolved Hide resolved

pkg/frontend/querymiddleware/spin_off_subqueries_test.go Show resolved Hide resolved

charleskorn approved these changes Jun 12, 2025

View reviewed changes

56quarters added 4 commits June 12, 2025 10:54

Make spin-off tests parallel

bffe858

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>

Code review feedback: support for fallback header in QFE

9438f50

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>

Code review feedback: Use the same tests for prom engine and MQE

d70fca9

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>

56quarters force-pushed the 56quarters/mqe-qfe branch from 8210b34 to d70fca9 Compare June 12, 2025 14:54

charleskorn reviewed Jun 12, 2025

View reviewed changes

pkg/mimir/modules.go Show resolved Hide resolved

Code review feedback: eagerly load selectors

920298a

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>

56quarters enabled auto-merge (squash) June 13, 2025 20:00

56quarters merged commit 0fc0f9a into main Jun 13, 2025
33 checks passed

56quarters deleted the 56quarters/mqe-qfe branch June 13, 2025 20:11

charleskorn mentioned this pull request Jun 17, 2025

MQE: emit query fallback stats from query-frontends #11775

Merged

1 task

charleskorn mentioned this pull request Jul 31, 2025

MQE: change log message to disambiguate from query-frontend query stats log messages #12251

Merged

1 task

56quarters mentioned this pull request Aug 1, 2025

chore: Fix configuration setting used in query-frontend error #12276

Merged

4 tasks

56quarters added a commit that referenced this pull request Aug 1, 2025

chore: Fix configuration setting used in query-frontend error (#12276)

9ef504c

Fix a copy/paste error from when MQE was added to the query-frontend. Related #11417

jesusvazquez pushed a commit that referenced this pull request Aug 2, 2025

chore: Fix configuration setting used in query-frontend error (#12276)

7cbea23

Fix a copy/paste error from when MQE was added to the query-frontend. Related #11417

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

mqe: Allow usage of Mimir Query Engine in query-frontend #11417

mqe: Allow usage of Mimir Query Engine in query-frontend #11417

Uh oh!

56quarters commented May 7, 2025 •

edited

Loading

Uh oh!

Uh oh!

charleskorn May 16, 2025

Uh oh!

56quarters Jun 5, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 5, 2025 •

edited

Loading

Uh oh!

charleskorn left a comment

Uh oh!

Uh oh!

Uh oh!

56quarters commented Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mqe: Allow usage of Mimir Query Engine in query-frontend #11417

mqe: Allow usage of Mimir Query Engine in query-frontend #11417

Uh oh!

Conversation

56quarters commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Which issue(s) this PR fixes or relates to

Notes to reviewers

Checklist

Uh oh!

Uh oh!

charleskorn May 16, 2025

Choose a reason for hiding this comment

Uh oh!

56quarters Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

charleskorn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

56quarters commented Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

56quarters commented May 7, 2025 •

edited

Loading

github-actions bot commented Jun 5, 2025 •

edited

Loading