You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As ingesters are restarted, their series are resharded onto other ingesters. During a rolling upgrade, this can lead to a lot of churn as series are resharded across virtually every ingester.
Since we know the ingesters in this chart are deployed via a StatefulSet, we know the hostname and storage is stable across restarts. This means we could feasibly run with the following settings:
If false, ingesters are not unregistered on shutdown and left in the ring with the LEAVING state. Setting to false prevents series resharding during ingesters rollouts, but requires to:
Either manually forget ingesters on scale down or invoke the /shutdown endpoint
Ensure ingester ID is preserved during rollouts
This is how we run Grafana Cloud, so it would make sense to me to bake this into the Helm Chart as well.