Skip to content

[mimir-distributed] Avoid series resharding during rolling ingester restart #1313

@Logiraptor

Description

@Logiraptor

As ingesters are restarted, their series are resharded onto other ingesters. During a rolling upgrade, this can lead to a lot of churn as series are resharded across virtually every ingester.

Since we know the ingesters in this chart are deployed via a StatefulSet, we know the hostname and storage is stable across restarts. This means we could feasibly run with the following settings:

-distributor.extend-writes=false
-ingester.unregister-on-shutdown=false

If false, ingesters are not unregistered on shutdown and left in the ring with the LEAVING state. Setting to false prevents series resharding during ingesters rollouts, but requires to:

  1. Either manually forget ingesters on scale down or invoke the /shutdown endpoint
  2. Ensure ingester ID is preserved during rollouts

This is how we run Grafana Cloud, so it would make sense to me to bake this into the Helm Chart as well.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions