[Bug]: TLS options for Prometheus are leading to crash when specified in a YAML configuration file #7364
Description
What happened?
Used image: jaegertracing/jaeger:2.8.0 (jaeger v2)
Deployed it using OpenTelemetryOperator via OpenTelemetryCollector resource, specified config in spec.config
Jaeger crashed with
'prometheus' has invalid keys: tls
passing CLI flag:
--set=extensions.jaeger_storage.metric_backends.prom.prometheus.tls.ca_file=/ca/ca.crt'
also not working and produced same output
Steps to reproduce
- Specify Prometheus storage backend with TLS options:
metric_backends:
prom:
prometheus:
endpoint: https://example.com
normalize_calls: true
normalize_duration: true
token_file_path: /tokens/prometheus_token
token_override_from_context: true
tls:
insecure: false
ca_file: /tls/ca.crt
- Deploy jaeger
- See logs
Expected behavior
Expected Prometheus TLS config to work just like in Cassandra storage config for example (since they are referring to the same struct go.opentelemetry.io/collector/config/configtls)
Relevant log output
'extensions' error reading configuration for "jaeger_storage": decoding failed due to the following error(s):
'metric_backends[prom]' decoding failed due to the following error(s):
'prometheus' has invalid keys: tlsScreenshot
No response
Additional context
Seems like the issue is coming from forgotten mapstructure at line 17 in file internal/config/promcfg/config.go:
// Configuration describes the options to customize the storage behavior.
type Configuration struct {
ServerURL string `valid:"required" mapstructure:"endpoint"`
ConnectTimeout time.Duration `mapstructure:"connect_timeout"`
TLS configtls.ClientConfig <----------------- here
TokenFilePath string `mapstructure:"token_file_path"`
TokenOverrideFromContext bool `mapstructure:"token_override_from_context"`
MetricNamespace string `mapstructure:"metric_namespace"`
LatencyUnit string `mapstructure:"latency_unit"`
NormalizeCalls bool `mapstructure:"normalize_calls"`
NormalizeDuration bool `mapstructure:"normalize_duration"`
// ExtraQueryParams is used to provide extra parameters to be appended
// to the URL of queries going out to the metrics backend.
ExtraQueryParams map[string]string `mapstructure:"extra_query_parameters"`
}
But I'm no golang expert
Jaeger backend version
2.8.0
SDK
I'm not sending any data
Pipeline
OTEL-SDK--->Jaeger all-in-one--->Scylladb
Stogage backend
Scylladb, VictoriaMetrics
Operating system
Linux
Deployment model
Kubernetes OpenTelemetry Operator
Deployment configs
# OpenTelemetryCollector spec.config:
config:
service:
telemetry:
resource:
service.name: jaeger-main
metrics:
readers:
- pull:
exporter:
prometheus:
host: 0.0.0.0
port: 8888
logs:
level: DEBUG
extensions: [jaeger_storage, jaeger_query]
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [jaeger_storage_exporter, spanmetrics]
metrics/spanmetrics:
receivers: [spanmetrics]
exporters: [prometheus]
connectors:
spanmetrics:
processors:
batch:
extensions:
jaeger_query:
storage:
traces: main
metrics: prom
jaeger_storage:
backends:
main:
cassandra:
schema:
create: true
datacenter: dc1
keyspace: jaeger
query:
timeout: 20s
consistency: TWO
connection:
servers: scylladb-0.scylladb-1.tracing.svc.cluster.local,scylladb-0.scylladb-2.tracing.svc.cluster.local,scylladb-0.scylladb-3.tracing.svc.cluster.local
tls:
insecure: false
ca_file: /scylladb-certs/ca.crt
# Otherwise jaeger will resolve hostnames first, and then will try to connect via IP (and it will break TLS because of no IPSans)
server_name_override: scylladb.tracing.svc.cluster.local
auth:
basic:
username: "jaeger"
password: "${env:CASSANDRA_PASSWORD}"
metric_backends:
prom:
prometheus:
endpoint: https://example.com
normalize_calls: true
normalize_duration: true
token_file_path: /tokens/prometheus_token
token_override_from_context: true
tls:
insecure: false
ca_file: /ca/ca.crt
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
jaeger_storage_exporter:
trace_storage: main
prometheus:
endpoint: "0.0.0.0:8889"