Prometheus 'Label value is not valid UTF-8' cause API timeout #2944
Description
Describe the bug
Prometheus 'Label value is not valid UTF-8' cause API request timeout.
To Reproduce
Steps to reproduce the behavior:
- docker run -d -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14268:14268 -p 14250:14250 -p 9411:9411 jaegertracing/all-in-one:1.22
- curl http://localhost:9411/%C0%AE%C0%AE
- Then all the other requests to http://localhost:9411/ will be timeout.
log:
2021/04/18 06:51:16 http: panic serving 172.17.0.1:33336: label value "/\xc0\xae\xc0\xae" is not valid UTF-8
goroutine 195 [running]:
net/http.(*conn).serve.func1(0xc000489f40)
net/http/server.go:1801 +0x147
panic(0x1392360, 0xc0002c3630)
runtime/panic.go:975 +0x47a
github.com/prometheus/client_golang/prometheus.(*HistogramVec).WithLabelValues(0xc0004fc500, 0xc000575770, 0x3, 0x3, 0xc000575650, 0xc000575770)
github.com/prometheus/client_golang@v1.5.1/prometheus/histogram.go:489 +0xda
github.com/uber/jaeger-lib/metrics/prometheus.(*Factory).Timer(0xc0000edc20, 0x15b70ed, 0x15, 0xc000575620, 0x15bbfcd, 0x19, 0x0, 0x0, 0x0, 0x1512b80, ...)
github.com/uber/jaeger-lib@v2.4.0+incompatible/metrics/prometheus/factory.go:181 +0x2ce
github.com/uber/jaeger-lib/metrics/fork.(*Factory).Timer(0xc0003fe870, 0x15b70ed, 0x15, 0xc000575620, 0x15bbfcd, 0x19, 0x0, 0x0, 0x0, 0xc0006799f5, ...)
github.com/uber/jaeger-lib@v2.4.0+incompatible/metrics/fork/fork.go:48 +0x65
github.com/jaegertracing/jaeger/pkg/httpmetrics.buildTimer(...)
github.com/jaegertracing/jaeger/pkg/httpmetrics/metrics.go:111
github.com/jaegertracing/jaeger/pkg/httpmetrics.(*requestDurations).record(0xc00056c090, 0xc0000bbc80, 0x3, 0xc00057a858, 0x5, 0xc00057a940, 0x3, 0x10578)
github.com/jaegertracing/jaeger/pkg/httpmetrics/metrics.go:101 +0x465
github.com/jaegertracing/jaeger/pkg/httpmetrics.Wrap.func1(0x1e1c2c0, 0xc000158380, 0xc00014ea00)
github.com/jaegertracing/jaeger/pkg/httpmetrics/metrics.go:63 +0x1fe
net/http.HandlerFunc.ServeHTTP(0xc00050de60, 0x1e1c2c0, 0xc000158380, 0xc00014ea00)
net/http/server.go:2042 +0x44
github.com/rs/cors.(*Cors).Handler.func1(0x1e1c2c0, 0xc000158380, 0xc00014ea00)
github.com/rs/cors@v1.7.0/cors.go:219 +0x1b9
net/http.HandlerFunc.ServeHTTP(0xc00050de80, 0x1e1c2c0, 0xc000158380, 0xc00014ea00)
net/http/server.go:2042 +0x44
net/http.serverHandler.ServeHTTP(0xc0000d0380, 0x1e1c2c0, 0xc000158380, 0xc00014ea00)
net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc000489f40, 0x1e225c0, 0xc00025a400)
net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
net/http/server.go:2969 +0x36c
Expected behavior
Other requests not timeout.
Screenshots
null
Version (please complete the following information):
- OS: MacOS & Linux
- Jaeger version: 1.22
- Deployment: Docker
What troubleshooting steps did you try?
The error is reported by Prometheus, so I add -e METRICS_BACKEND=expvar to disable Prometheus.
Additional context
null