What is the bug?

Hi Team,

We are seeing cases where histogram_count are giving negative values intermittently.

We also recently upgraded to 2.16.0 to see if that alleviates the issue as it includes fix related to aggr_over_time and counter reset detection but issue still persists.

How to reproduce it?

Start Mimir 2.16.0
One of rule is based on query avg_over_time(histogram_count(sum(rate(query:without_instance{bushido_mode="shogun",bushido_tier=~"production.*"}[5m])))[5m:]) which trigger alerts with a negative value

However, when we visualize the same query in a dashboard it shows expected values (non-negative), so it's only seen during rule evaluation.

What did you think would happen?

histogram_count should never emit negative values

What was your environment?

Kubernetes

Any additional context to share?

No response