Prometheus panic with scraping azure_sd_configs #16466
Description
Hi Prometheus team,
this issue was observed after discussion from prometheus-community/helm-charts#5538. I'm using Workload Identity to scrape some VMSS from Azure from my Prometheus which is running in my AKS. I'm using the kube-prometheus-stack with the additional values.yaml:
prometheus:
serviceAccount:
annotations:
azure.workload.identity/client-id: client id of a dedicated Managed Identity
prometheusSpec:
podMetadata:
labels:
azure.workload.identity/use: "true"
additionalScrapeConfigs:
- job_name: azure-vm-monitor
azure_sd_configs:
- authentication_method: SDK
subscription_id: XXX
resource_group: XXX
port: 9100
refresh_interval: 30sThe Prometheus pod is crashing after the scraping of the job azure-vm-monitor (also it counts 0 VMs, but there are some instances from the VMSS active):
prometheus time=2025-04-22T03:21:36.273Z level=DEBUG source=azure.go:356 msg="Found virtual machines during Azure discovery." component="discovery manager scrape" discovery=azure config=azure-vm-monitor count=0
prometheus panic: Value kind is Float64, not Time
prometheus
prometheus goroutine 1535 [running]:
prometheus log/slog.Value.Time({{}, 0x0?, {0x350e9a0?, 0x4805280?}})
prometheus /usr/local/go/src/log/slog/value.go:379 +0x168
prometheus github.com/prometheus/common/promslog.init.func2({0xc0033035c0?, 0x350e9a0?, 0x4805280?}, {{0x3ea6d14, 0x4}, {{}, 0x4074900000000000, {0x350e9a0, 0x4805280}}})
prometheus /go/pkg/mod/github.com/prometheus/common@v0.62.0/promslog/slog.go:75 +0x90
prometheus log/slog.(*handleState).appendAttr(0xc0033035c0, {{0x3ea6d14, 0x4}, {{}, 0x4074900000000000, {0x350e9a0, 0x4805280}}})
prometheus /usr/local/go/src/log/slog/handler.go:475 +0x21e
prometheus log/slog.(*handleState).appendNonBuiltIns.func1({{0x3ea6d14, 0x4}, {{}, 0x4074900000000000, {0x350e9a0, 0x4805280}}})
prometheus /usr/local/go/src/log/slog/handler.go:342 +0x52
prometheus log/slog.Record.Attrs({{0xc1f9e0b02580f646, 0x195fe2c10, 0x6a10d60}, {0x3eb2367, 0xa}, 0xfffffffffffffffc, 0x144dd83, {{{0x3ea66aa, 0x3}, {{...}, ...}}, ...}, ...}, ...)
prometheus /usr/local/go/src/log/slog/record.go:84 +0xaa
prometheus log/slog.(*handleState).appendNonBuiltIns(_, {{0xc1f9e0b02580f646, 0x195fe2c10, 0x6a10d60}, {0x3eb2367, 0xa}, 0xfffffffffffffffc, 0x144dd83, {{{0x3ea66aa, 0x3}, ...}, ...}, ...})
prometheus /usr/local/go/src/log/slog/handler.go:341 +0x4c7
prometheus log/slog.(*commonHandler).handle(_, {{0xc1f9e0b02580f646, 0x195fe2c10, 0x6a10d60}, {0x3eb2367, 0xa}, 0xfffffffffffffffc, 0x144dd83, {{{0x3ea66aa, 0x3}, ...}, ...}, ...})
prometheus /usr/local/go/src/log/slog/handler.go:308 +0x5a5
prometheus log/slog.(*TextHandler).Handle(_, {_, _}, {{0xc1f9e0b02580f646, 0x195fe2c10, 0x6a10d60}, {0x3eb2367, 0xa}, 0xfffffffffffffffc, 0x144dd83, ...})
prometheus /usr/local/go/src/log/slog/text_handler.go:95 +0x3f
prometheus log/slog.(*Logger).log(0xc01c8e25d0, {0x485de28, 0x6a35580}, 0xfffffffffffffffc, {0x3eb2367, 0xa}, {0xc003303bd0, 0x4, 0x4})
prometheus /usr/local/go/src/log/slog/logger.go:257 +0x1e4
prometheus log/slog.(*Logger).Debug(...)
prometheus /usr/local/go/src/log/slog/logger.go:200
prometheus github.com/prometheus/prometheus/discovery/azure.(*Discovery).addToCache(0xc01c8ef260, {0xc01cdbc870, 0xe7}, 0xc000b18680)
prometheus /app/discovery/azure/azure.go:727 +0x24b
prometheus github.com/prometheus/prometheus/discovery/azure.(*Discovery).vmToLabelSet(0xc01c8ef260, {0x485ded0, 0xc01cabe730}, {0x4863ce0, 0xc019499500}, {{0xc011b71790, 0xcc}, {0xc002e8ea68, 0x18}, {0xc01ff75aa0, ...}, ...})
prometheus /app/discovery/azure/azure.go:471 +0x78e
prometheus github.com/prometheus/prometheus/discovery/azure.(*Discovery).refreshAzureClient.func1({{0xc011b71790, 0xcc}, {0xc002e8ea68, 0x18}, {0xc01ff75aa0, 0x1a}, {0xc002dfd0c0, 0x39}, {0xc002e8e8a0, 0x12}, ...})
prometheus /app/discovery/azure/azure.go:387 +0xbe
prometheus created by github.com/prometheus/prometheus/discovery/azure.(*Discovery).refreshAzureClient in goroutine 700
prometheus /app/discovery/azure/azure.go:385 +0x546
The authentication for the used Managed Identity is working, as I can see in the logs of Azure Entra ID. The identity has reader permission on the defined resource group.
There are some tags from Azure DevOps:
__AzureDevOpsElasticPool : Azure VMSS Linux Pipelines
__AzureDevOpsElasticPoolTimeStamp : 4/22/2025 8:48:55 AM
Maybe those tags are the problem?