Skip to content

Fix panic at startup when Mimir is running in monolithic mode and query sharding is enabled #2036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 7, 2022
Merged

Fix panic at startup when Mimir is running in monolithic mode and query sharding is enabled #2036

merged 1 commit into from
Jun 7, 2022

Conversation

pracucci
Copy link
Collaborator

@pracucci pracucci commented Jun 7, 2022

What this PR does

I'm doing some tests and I've just realized that Mimir panics at startup when running in monolithic mode and query sharding is enabled, because of conflicting metrics registered by PromQL engine:

panic: a previously registered descriptor with the same fully-qualified name as Desc{fqName: "prometheus_engine_queries", help: "The current number of queries being executed or waiting.", constLabels: {engine="querier"}, variableLabels: []} has different label names or a different help string

goroutine 1 [running]:
github.com/prometheus/client_golang/prometheus.(*wrappingRegisterer).MustRegister(0xc010ee4f90, {0xc01102c000, 0x5, 0x6})
	/go/src/github.com/grafana/mimir/vendor/github.com/prometheus/client_golang/prometheus/wrap.go:104 +0x151
github.com/prometheus/prometheus/promql.NewEngine({{0x235b9a0, 0xc00036aaa0}, {0x237bed0, 0xc010ee4f90}, 0x2faf080, 0x45d964b800, {0x237b510, 0xc010ec0f00}, 0x45d964b800, 0xc0005d37d0, ...})
	/go/src/github.com/grafana/mimir/vendor/github.com/prometheus/prometheus/promql/engine.go:350 +0x8f0
github.com/grafana/mimir/pkg/querier.New({0x0, 0x1, 0x2a9079602000, 0x274a48a78000, 0x8bb2c97000, {0x0, {{0x0, 0x0}, {0x0, 0x0}, ...}}, ...}, ...)
	/go/src/github.com/grafana/mimir/pkg/querier/querier.go:134 +0x3a5
github.com/grafana/mimir/pkg/mimir.(*Mimir).initQueryable(0xc00062e000)
	/go/src/github.com/grafana/mimir/pkg/mimir/modules.go:311 +0x1c5
github.com/grafana/dskit/modules.(*Manager).initModule(0xc00000ce88, {0x7ffef8c6a608, 0x1e16cab}, 0x1e, 0x0)
	/go/src/github.com/grafana/mimir/vendor/github.com/grafana/dskit/modules/modules.go:120 +0x224
github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0x1aee9c0, {0xc00093e960, 0x1, 0x4})
	/go/src/github.com/grafana/mimir/vendor/github.com/grafana/dskit/modules/modules.go:92 +0x10c
github.com/grafana/mimir/pkg/mimir.(*Mimir).Run(0xc00062e000)
	/go/src/github.com/grafana/mimir/pkg/mimir/mimir.go:438 +0x21c
main.main()
	/go/src/github.com/grafana/mimir/cmd/mimir/main.go:204 +0xa90

This PR fixes it, adding engine label to PromQL engine registerer used by query-frontend. The engine label is already injected by querier and ruler, so in this PR I'm just following what we already did there.

Which issue(s) this PR fixes or relates to

N/A

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@pracucci
Fix panic at startup when Mimir is running in monolithic mode and que…
1157744 
…ry sharding is enabled

Signed-off-by: Marco Pracucci <marco@pracucci.com>
@pracucci pracucci merged commit 66fb82f into main Jun 7, 2022
@pracucci pracucci deleted the fix-query-sharding-in-monolithic-mode branch June 7, 2022 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pstibrany pstibrany pstibrany approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pracucci @pstibrany