OTel Metrics Receiver does not process all resource attributes #13192
Description
What did you do?
(The sample code and commit with details is in another commit, but I will summarise here)
Recently, I've been writing a course teaching people how to reason through time series, and I discovered to my happiness that Prometheus supports OpenTelemetry Write. This makes my life much easier from a teaching perspective, as it reduces the number of concepts I need to teach 🎉
While experimenting with this, I noticed that while Prometheus converts the semantic convention service.name into job, it doesn't convert all resource-supplied attributes. The code is here:
prometheus/storage/remote/otlptranslator/prometheusremotewrite/helper.go
Lines 161 to 162 in 965e603
And checks for two specific instances of attributes, or those defined with the SDK Default value.
What did you expect to see?
I tried to add "hostname" (via semconv.Hostname) to the attribute set, but it got dropped. I was confused, and went digging until I found the code linked above to explain it. The resource code is just:
ctx := context.Background()
hostname, _ := os.Hostname()
// Setup the "application resource". This is the way in which the application is identified in OpenTelemetry,
// as well as any "core attributes" it has.
res, err := resource.New(ctx, resource.WithAttributes(
semconv.ServiceName(applicationName),
semconv.HostName(hostname),
))
if err != nil {
return nil, nil, fmt.Errorf("%w: %s", ErrFailedMeterSetup, err)
}
You can see the stdout exported version below; both "custom" and "host.name" are dropped.
{
"Resource": [
{
"Key": "host.name",
"Value": {
"Type": "STRING",
"Value": "c33dc"
}
},
{
"Key": "service.name",
"Value": {
"Type": "STRING",
"Value": "counter_simple"
}
}
],
"ScopeMetrics": [
{
"Scope": {
"Name": "pito.local/examples/metrics/counter_simple",
"Version": "",
"SchemaURL": ""
},
"Metrics": [
{
"Name": "coffees",
"Description": "",
"Unit": "",
"Data": {
"DataPoints": [
{
"Attributes": [],
"StartTime": "2023-11-27T11:23:30.570441387+01:00",
"Time": "2023-11-27T11:23:30.570449179+01:00",
"Value": 3
}
],
"Temporality": "CumulativeTemporality",
"IsMonotonic": true
}
}
]
}
]
}
What did you see instead? Under which circumstances?
The attributes. Other vendors (e.g. Lightstep) do this, and I think companies tend to add attributes to these as a kind of, "default set" indicating deployment version and so on.
System information
Linux 6.1.0-13-amd64 x86_64
Prometheus version
prometheus, version 2.48.0 (branch: HEAD, revision: 6d80b30990bc297d95b5c844e118c4011fad8054)
build user: root@26117804242c
build date: 20231116-04:35:21
go version: go1.21.4
platform: linux/amd64
tags: netgo,builtinassets,stringlabels
Prometheus configuration file
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: "prometheus"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ["localhost:9090"]
- job_name: "node-exporter"
static_configs:
- targets: ["localhost:9100"]Alertmanager version
None
Alertmanager configuration file
NoneLogs
None