Skip to content

Native histograms: appending valid histograms corrupts chunks #16894

@krajorama

Description

@krajorama

What did you do?

We were testing a new feature at Grafana that was emitting some useless , but valid native histograms that had many empty buckets at random indexes.

What did you expect to see?

No errors on readback.

What did you see instead? Under which circumstances?

Panic when executing a simple sum(rate(...)) query.

panic: runtime error: index out of range [-1] [recovered]
	panic: runtime error: index out of range [-1]

goroutine 182182 [running]:
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.deferwrap1()
	/__w/backend-enterprise/backend-enterprise/vendor/go.opentelemetry.io/otel/sdk/trace/span.go:467 +0x2c
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0x401ec3a3c0, {0x0, 0x0, 0x89714?})
	/__w/backend-enterprise/backend-enterprise/vendor/go.opentelemetry.io/otel/sdk/trace/span.go:506 +0xa04
panic({0x372ec60?, 0x4041de85a0?})
	$GOROOT/src/runtime/panic.go:792 +0x124
github.com/prometheus/prometheus/model/histogram.addBuckets(0x0, 0x37f0000000000000, 0x1, {0x40304f3690?, 0x4026742780?, 0x24ff084?}, {0x40304f36a0?, 0x19815d8653f?, 0x19815d8da6f?}, {0x405a5a8cf0, ...}, ...)
	/__w/backend-enterprise/backend-enterprise/vendor/github.com/prometheus/prometheus/model/histogram/float_histogram.go:1270 +0x998
github.com/prometheus/prometheus/model/histogram.(*FloatHistogram).Sub(0x40270dd860, 0x4026723ea0)
	/__w/backend-enterprise/backend-enterprise/vendor/github.com/prometheus/prometheus/model/histogram/float_histogram.go:456 +0x3f0
github.com/grafana/mimir/pkg/streamingpromql/operators/functions.histogramRate(0x0, 0x18, {0x402117ce58?, 0x18, 0x4026732c88?}, {0x0, 0x0, 0x0}, 0x19815c984a8, 0x19815d77ae0, ...)

System information

No response

Prometheus version


Prometheus configuration file

Alertmanager version


Alertmanager configuration file

Logs


Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions