-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Description
github.com/prometheus/client_golang v1.16.0
Hi 👋
I am using the latest golang client to expose metrics using exemplars, and I have encountered a case in which the client tolerates providing an empty labels map when observing with exemplars.
This results in prometheus failing to scrape because the metrics format is invalid.
A code sample that generates an invalid metric would look like this:
package main
import (
"log"
"net/http"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
func main() {
t := time.Now()
failedHistogram := promauto.NewHistogramVec(prometheus.HistogramOpts{
Namespace: "prometheus",
Subsystem: "poc",
Name: "test",
Buckets: prometheus.DefBuckets,
Help: "my help",
}, []string{"result"})
observer := failedHistogram.WithLabelValues("failed")
withExemplar, ok := observer.(prometheus.ExemplarObserver)
if !ok {
panic("could not use observer as an exemplar observer")
}
// The following line would yield a valid scrape
// withExemplar.ObserveWithExemplar(time.Since(t).Seconds(), prometheus.Labels{"traceID": "1"})
// The following line is allowed by the client, yet renders an invalid metrics page
withExemplar.ObserveWithExemplar(time.Since(t).Seconds(), prometheus.Labels{})
log.Println("Listening on port 8000")
http.ListenAndServe(":8000", promhttp.InstrumentMetricHandler(
prometheus.DefaultRegisterer,
promhttp.HandlerFor(prometheus.DefaultGatherer,
promhttp.HandlerOpts{
// Necessary to enable exemplars
EnableOpenMetrics: true,
}),
))
}
On scrape I get an error on the target: expected timestamp or # symbol, got "INVALID"
which hasn't improved a lot upgrading prometheus to the latest (2.46.0) as it only adds the metric name where it's failing.
Upon further testing scraping with curl (curl -H 'Accept: application/openmetrics-text' localhost:8000
), I cornered the problem to it being that ObserveWithExemplar tolerates receiving an empty prometheus.Labels{}
which in turn fails to render, examples:
Failed scrape (mind the double space after #
):
prometheus_poc_test_bucket{result="failed",le="0.005"} 1 # 5.8e-05 1.692205065685987e+09
or when there is a label, successful scrape:
prometheus_poc_test_bucket{result="failed",le="0.005"} 1 # {traceID="1"} 4.4125e-05 1.692206618903436e+09
I think that either ObserveWithExemplar should blow up when there's an empty label, or this should be ignored silently.
BTW, Go mod looks like this:
module github.com/pcarranza/prometheus_poc
go 1.20
require github.com/prometheus/client_golang v1.16.0
require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/prometheus/client_model v0.3.0 // indirect
github.com/prometheus/common v0.42.0 // indirect
github.com/prometheus/procfs v0.10.1 // indirect
golang.org/x/sys v0.8.0 // indirect
google.golang.org/protobuf v1.30.0 // indirect
)