-
Notifications
You must be signed in to change notification settings - Fork 9.8k
Closed
Milestone
Description
What did you do?
We were testing a new feature at Grafana that was emitting some useless , but valid native histograms that had many empty buckets at random indexes.
What did you expect to see?
No errors on readback.
What did you see instead? Under which circumstances?
Panic when executing a simple sum(rate(...))
query.
panic: runtime error: index out of range [-1] [recovered]
panic: runtime error: index out of range [-1]
goroutine 182182 [running]:
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.deferwrap1()
/__w/backend-enterprise/backend-enterprise/vendor/go.opentelemetry.io/otel/sdk/trace/span.go:467 +0x2c
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0x401ec3a3c0, {0x0, 0x0, 0x89714?})
/__w/backend-enterprise/backend-enterprise/vendor/go.opentelemetry.io/otel/sdk/trace/span.go:506 +0xa04
panic({0x372ec60?, 0x4041de85a0?})
$GOROOT/src/runtime/panic.go:792 +0x124
github.com/prometheus/prometheus/model/histogram.addBuckets(0x0, 0x37f0000000000000, 0x1, {0x40304f3690?, 0x4026742780?, 0x24ff084?}, {0x40304f36a0?, 0x19815d8653f?, 0x19815d8da6f?}, {0x405a5a8cf0, ...}, ...)
/__w/backend-enterprise/backend-enterprise/vendor/github.com/prometheus/prometheus/model/histogram/float_histogram.go:1270 +0x998
github.com/prometheus/prometheus/model/histogram.(*FloatHistogram).Sub(0x40270dd860, 0x4026723ea0)
/__w/backend-enterprise/backend-enterprise/vendor/github.com/prometheus/prometheus/model/histogram/float_histogram.go:456 +0x3f0
github.com/grafana/mimir/pkg/streamingpromql/operators/functions.histogramRate(0x0, 0x18, {0x402117ce58?, 0x18, 0x4026732c88?}, {0x0, 0x0, 0x0}, 0x19815c984a8, 0x19815d77ae0, ...)
System information
No response
Prometheus version
Prometheus configuration file
Alertmanager version
Alertmanager configuration file
Logs