Skip to content

Bug: empty exemplar labels results in invalid metrics format #1333

@pcarranza

Description

@pcarranza

github.com/prometheus/client_golang v1.16.0

Hi 👋

I am using the latest golang client to expose metrics using exemplars, and I have encountered a case in which the client tolerates providing an empty labels map when observing with exemplars.

This results in prometheus failing to scrape because the metrics format is invalid.

A code sample that generates an invalid metric would look like this:

package main

import (
	"log"
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
	t := time.Now()

	failedHistogram := promauto.NewHistogramVec(prometheus.HistogramOpts{
		Namespace: "prometheus",
		Subsystem: "poc",
		Name:      "test",
		Buckets:   prometheus.DefBuckets,
		Help:      "my help",
	}, []string{"result"})

	observer := failedHistogram.WithLabelValues("failed")

	withExemplar, ok := observer.(prometheus.ExemplarObserver)
	if !ok {
		panic("could not use observer as an exemplar observer")
	}

	// The following line would yield a valid scrape 
	// withExemplar.ObserveWithExemplar(time.Since(t).Seconds(), prometheus.Labels{"traceID": "1"})

	// The following line is allowed by the client, yet renders an invalid metrics page
	withExemplar.ObserveWithExemplar(time.Since(t).Seconds(), prometheus.Labels{})

	log.Println("Listening on port 8000")
	http.ListenAndServe(":8000", promhttp.InstrumentMetricHandler(
		prometheus.DefaultRegisterer,
		promhttp.HandlerFor(prometheus.DefaultGatherer,
			promhttp.HandlerOpts{
				// Necessary to enable exemplars
				EnableOpenMetrics: true,
			}),
	))
}

On scrape I get an error on the target: expected timestamp or # symbol, got "INVALID" which hasn't improved a lot upgrading prometheus to the latest (2.46.0) as it only adds the metric name where it's failing.

Upon further testing scraping with curl (curl -H 'Accept: application/openmetrics-text' localhost:8000), I cornered the problem to it being that ObserveWithExemplar tolerates receiving an empty prometheus.Labels{} which in turn fails to render, examples:

Failed scrape (mind the double space after #):

prometheus_poc_test_bucket{result="failed",le="0.005"} 1 #  5.8e-05 1.692205065685987e+09

or when there is a label, successful scrape:

prometheus_poc_test_bucket{result="failed",le="0.005"} 1 # {traceID="1"} 4.4125e-05 1.692206618903436e+09

I think that either ObserveWithExemplar should blow up when there's an empty label, or this should be ignored silently.

BTW, Go mod looks like this:

module github.com/pcarranza/prometheus_poc

go 1.20

require github.com/prometheus/client_golang v1.16.0

require (
	github.com/beorn7/perks v1.0.1 // indirect
	github.com/cespare/xxhash/v2 v2.2.0 // indirect
	github.com/golang/protobuf v1.5.3 // indirect
	github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
	github.com/prometheus/client_model v0.3.0 // indirect
	github.com/prometheus/common v0.42.0 // indirect
	github.com/prometheus/procfs v0.10.1 // indirect
	golang.org/x/sys v0.8.0 // indirect
	google.golang.org/protobuf v1.30.0 // indirect
)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions