OpenMetrics using gigabytes more memory #14808
Description
What did you do?
Ran Prometheus at commit a77f500
What did you expect to see?
Heap memory around 15-20GB
What did you see instead? Under which circumstances?
OOM at 32GB limit; also CPU usage went way up.
Going back to commit b7a58dc resolved the problem (before #14356)
System information
Linux 6.1.85+ #1 SMP PREEMPT_DYNAMIC Thu Jun 27 21:05:47 UTC 2024 x86_64
Prometheus version
prometheus, version 2.54.1 (branch: main, revision: a77f5007f9debcc1fc5b7133ac422e2b164a8c99)
build user: bryan@study-pc
build date: 20240902-11:24:22
go version: go1.22.5
platform: linux/amd64
tags: netgo,builtinassets,stringlabels
Prometheus configuration file
Command-line args:
- args:
- --config.file=/etc/prometheus/$(POD_NAME).yml
- --web.listen-address=:9090
- --web.external-url=https://admin-dev-us-central-0.grafana-dev.net/bryan-prometheus/
- --web.enable-admin-api
- --web.enable-lifecycle
- --web.route-prefix=/bryan-prometheus/
- --storage.tsdb.path=/prometheus/data
- --storage.tsdb.wal-compression
- --storage.tsdb.retention.size=40GB
- --enable-feature=promql-at-modifier,exemplar-storage
Alertmanager version
No response
Alertmanager configuration file
No response
Logs
ts=2024-09-02T12:10:47.610Z caller=main.go:252 level=warn msg="This option for --enable-feature is now permanently enabled and therefore a no-op." option=promql-at-modifier
ts=2024-09-02T12:10:47.610Z caller=main.go:193 level=info msg="Experimental in-memory exemplar storage enabled"
ts=2024-09-02T12:10:47.614Z caller=main.go:671 level=info msg="Starting Prometheus Server" mode=server version="(version=2.54.1, branch=main, revision=a77f5007f9debcc1fc5b7133ac422e2b164a8c99)"
ts=2024-09-02T12:10:47.614Z caller=main.go:676 level=info build_context="(go=go1.22.5, platform=linux/amd64, user=bryan@study-pc, date=20240902-11:24:22, tags=netgo,builtinassets,stringlabels)"
ts=2024-09-02T12:10:47.614Z caller=main.go:677 level=info host_details="(Linux 6.1.85+ #1 SMP PREEMPT_DYNAMIC Thu Jun 27 21:05:47 UTC 2024 x86_64 bryan-prometheus-0 (none))"
ts=2024-09-02T12:10:47.614Z caller=main.go:678 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2024-09-02T12:10:47.614Z caller=main.go:679 level=info vm_limits="(soft=unlimited, hard=unlimited)"