Skip to content

fatal error: sync: unlock of unlocked mutex #16318

New issue
New issue
Closed
#16332
Closed
fatal error: sync: unlock of unlocked mutex#16318
#16332
Assignees
Alhanaqtah
Labels
kind/bug
@dbluxo

Description

@dbluxo
dbluxo
opened on Mar 25, 2025

What did you do?

I have updated Prometheus from v2.55.0 to v3.2.1 with the feature created-timestamp-zero-ingestion enabled.

What did you expect to see?

Successful update and start-up of Prometheus.

What did you see instead? Under which circumstances?

Prometheus failed to start successfully. And logs fatal error: sync: unlock of unlocked mutex. If I disable the feature created-timestamp-zero-ingestion, Prometheus starts successfully, if I enable it again, the start fails again.

System information

Linux 5.14.0-284.104.1.el9_2.x86_64 x86_64

Prometheus version

prometheus, version 3.2.1 (branch: HEAD, revision: 804c49d58f3f3784c77c9c8ec17c9062092cae27)
  build user:       root@42fd9529152a
  build date:       20250226-09:19:49
  go version:       go1.23.6
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

Prometheus configuration file



Alertmanager version






Alertmanager configuration file




Logs


time=2025-03-25T08:48:35.432Z level=INFO source=main.go:256 msg="Experimental native histogram support enabled. Changed default scrape_protocols to prefer PrometheusProto format." global.scrape_protocols="[PrometheusProto OpenMetricsText1.0.0 OpenMetricsText0.0.1 PrometheusText1.0.0 PrometheusText0.0.4]"
time=2025-03-25T08:48:35.432Z level=INFO source=main.go:266 msg="Experimental created timestamp zero ingestion enabled. Changed default scrape_protocols to prefer PrometheusProto format." global.scrape_protocols="[PrometheusProto OpenMetricsText1.0.0 OpenMetricsText0.0.1 PrometheusText1.0.0 PrometheusText0.0.4]"
time=2025-03-25T08:48:35.432Z level=INFO source=main.go:225 msg="Experimental in-memory exemplar storage enabled"
time=2025-03-25T08:48:35.434Z level=INFO source=main.go:687 msg="Starting Prometheus Server" mode=server version="(version=3.2.1, branch=HEAD, revision=804c49d58f3f3784c77c9c8ec17c9062092cae27)"
time=2025-03-25T08:48:35.434Z level=INFO source=main.go:692 msg="operational information" build_context="(go=go1.23.6, platform=linux/amd64, user=root@42fd9529152a, date=20250226-09:19:49, tags=netgo,builtinassets,stringlabels)" host_details="(Linux 5.14.0-284.104.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Feb 10 10:00:03 EST 2025 x86_64 prometheus-server-1 (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2025-03-25T08:48:35.435Z level=INFO source=main.go:768 msg="Updating GOMAXPROCS=2: determined from CPU quota" component=automaxprocs
time=2025-03-25T08:48:35.437Z level=INFO source=web.go:654 msg="Start listening for connections" component=web address=0.0.0.0:9090
time=2025-03-25T08:48:35.437Z level=INFO source=main.go:1228 msg="Starting TSDB ..."
time=2025-03-25T08:48:35.438Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1710325782550 maxt=1711108800000 ulid=01HSKHPA60JC7CQ2GY3NEVT43T
time=2025-03-25T08:48:35.438Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1711108800023 maxt=1712858400000 ulid=01HV7P86XSMFVK2R9JP0NGF6X2
time=2025-03-25T08:48:35.438Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1712858401251 maxt=1714608000000 ulid=01HWVTS7MJQN3CNF4KJTQ9GN6A
time=2025-03-25T08:48:35.438Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1714608001251 maxt=1716357600000 ulid=01HYFZAV5E4R4ZW73R37BGMD7P
time=2025-03-25T08:48:35.438Z level=INFO source=tls_config.go:347 msg="Listening on" component=web address=[::]:9090
time=2025-03-25T08:48:35.439Z level=INFO source=tls_config.go:350 msg="TLS is disabled." component=web http2=false address=[::]:9090
time=2025-03-25T08:48:35.438Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1716357600030 maxt=1718107200000 ulid=01J043WB13ZDS8AFT79TEKMZDA
time=2025-03-25T08:48:35.439Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1718107200463 maxt=1719856800000 ulid=01J1R8DVN9BFXJ4DWWS4NYR9MY
time=2025-03-25T08:48:35.439Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1719856800113 maxt=1721606400000 ulid=01J3CCZE75K8693JEXAMPC1BHS
time=2025-03-25T08:48:35.439Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1721606400787 maxt=1723356000000 ulid=01J50HH1B89S3H0PRGXPG3AY01
time=2025-03-25T08:48:35.439Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1723356001251 maxt=1725105600000 ulid=01J6MP2H7S66K0MH24E8ZZ7CEV
time=2025-03-25T08:48:35.439Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1725105600178 maxt=1726855200000 ulid=01J88TM4PCRTN9DKPK603HZFJJ
time=2025-03-25T08:48:35.440Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1726855200344 maxt=1728604800000 ulid=01J9WZ5XH7TMYD22EZWRC905K2
time=2025-03-25T08:48:35.440Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1728604800465 maxt=1730354400000 ulid=01JBH3Q8J5V2M5YZY845XX7WZ0
time=2025-03-25T08:48:35.440Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1730354400560 maxt=1732104000000 ulid=01JD588TJDAWYXFJAW4ZXHTZ8B
time=2025-03-25T08:48:35.440Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732104000817 maxt=1733853600000 ulid=01JESCTA6Q0H09QQXSHF2DMTF4
time=2025-03-25T08:48:35.440Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1733853600857 maxt=1735603200000 ulid=01JGDHBV4JDHSVTWX5DMEWHV1G
time=2025-03-25T08:48:35.440Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1735603200024 maxt=1737352800000 ulid=01JJ1NXFWGZAT2V3RKH7F1GRAQ
time=2025-03-25T08:48:35.441Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1737352800059 maxt=1739102400000 ulid=01JKNTEZKFQ27VAW58E8PYJQGY
time=2025-03-25T08:48:35.441Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1739102401250 maxt=1740852000000 ulid=01JN9Z0HK6NRNNEKKQQ3VGEMAT
time=2025-03-25T08:48:35.441Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1740852000954 maxt=1742601600000 ulid=01JPY3J4DD7DJW2GS9FZZ8M8P6
time=2025-03-25T08:48:35.441Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1742601601251 maxt=1742796000000 ulid=01JQ3WYH3HZZRWW5MCEY6Q4JFR
time=2025-03-25T08:48:35.441Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1742860800000 maxt=1742868000000 ulid=01JQ5M1KZEWVH7SMYGQ7K297RY
time=2025-03-25T08:48:35.441Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1742868000000 maxt=1742875200000 ulid=01JQ5V72ZCSWQ8VZNXHVC1HQ01
time=2025-03-25T08:48:35.441Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1742796001251 maxt=1742860800000 ulid=01JQ5V736TZHHAT2XE6A0F61KY
time=2025-03-25T08:48:35.441Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1742875200000 maxt=1742882400000 ulid=01JQ61SV5X1042SDBHVVYS0HXZ
time=2025-03-25T08:48:35.441Z level=WARN source=dir_locker.go:76 msg="A lockfile from a previous execution already existed. It was replaced" component=tsdb file=/data/lock
time=2025-03-25T08:48:35.501Z level=INFO source=head.go:628 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2025-03-25T08:48:35.501Z level=INFO source=head.go:715 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=5.331µs
time=2025-03-25T08:48:35.501Z level=INFO source=head.go:723 msg="Replaying WAL, this may take a while" component=tsdb
time=2025-03-25T08:48:35.673Z level=INFO source=head.go:760 msg="WAL checkpoint loaded" component=tsdb
time=2025-03-25T08:48:35.695Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4761 maxSegment=4794
time=2025-03-25T08:48:35.695Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4762 maxSegment=4794
time=2025-03-25T08:48:35.696Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4763 maxSegment=4794
time=2025-03-25T08:48:35.697Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4764 maxSegment=4794
time=2025-03-25T08:48:35.697Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4765 maxSegment=4794
time=2025-03-25T08:48:35.697Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4766 maxSegment=4794
time=2025-03-25T08:48:35.698Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4767 maxSegment=4794
time=2025-03-25T08:48:35.699Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4768 maxSegment=4794
time=2025-03-25T08:48:35.710Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4769 maxSegment=4794
time=2025-03-25T08:48:35.813Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4770 maxSegment=4794
time=2025-03-25T08:48:35.814Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4771 maxSegment=4794
time=2025-03-25T08:48:35.815Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4772 maxSegment=4794
time=2025-03-25T08:48:35.815Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4773 maxSegment=4794
time=2025-03-25T08:48:35.815Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4774 maxSegment=4794
time=2025-03-25T08:48:35.816Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4775 maxSegment=4794
time=2025-03-25T08:48:35.827Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4776 maxSegment=4794
time=2025-03-25T08:48:35.838Z level=WARN source=head_wal.go:423 msg="Unknown series references" component=tsdb samples=0 exemplars=1 histograms=0 metadata=0
time=2025-03-25T08:48:35.838Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4777 maxSegment=4794
time=2025-03-25T08:48:35.839Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4778 maxSegment=4794
time=2025-03-25T08:48:35.851Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4779 maxSegment=4794
time=2025-03-25T08:48:35.851Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4780 maxSegment=4794
time=2025-03-25T08:48:35.852Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4781 maxSegment=4794
time=2025-03-25T08:48:35.852Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4782 maxSegment=4794
time=2025-03-25T08:48:35.853Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4783 maxSegment=4794
time=2025-03-25T08:48:35.853Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4784 maxSegment=4794
time=2025-03-25T08:48:35.858Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4785 maxSegment=4794
time=2025-03-25T08:48:35.859Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4786 maxSegment=4794
time=2025-03-25T08:48:35.860Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4787 maxSegment=4794
time=2025-03-25T08:48:35.860Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4788 maxSegment=4794
time=2025-03-25T08:48:35.860Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4789 maxSegment=4794
time=2025-03-25T08:48:35.862Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4790 maxSegment=4794
time=2025-03-25T08:48:35.863Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4791 maxSegment=4794
time=2025-03-25T08:48:35.866Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4792 maxSegment=4794
time=2025-03-25T08:48:35.869Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4793 maxSegment=4794
time=2025-03-25T08:48:35.874Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=4794 maxSegment=4794
time=2025-03-25T08:48:35.874Z level=INFO source=head.go:832 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=172.429182ms wal_replay_duration=200.253033ms wbl_replay_duration=175ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=5.331µs total_replay_duration=372.717445ms
time=2025-03-25T08:48:36.012Z level=INFO source=main.go:1249 msg="filesystem information" fs_type=EXT4_SUPER_MAGIC
time=2025-03-25T08:48:36.012Z level=INFO source=main.go:1252 msg="TSDB started"
time=2025-03-25T08:48:36.012Z level=INFO source=main.go:1437 msg="Loading configuration file" filename=/etc/config/prometheus.yml
time=2025-03-25T08:48:36.014Z level=INFO source=kubernetes.go:323 msg="Using pod service account via in-cluster config" component="discovery manager scrape" discovery=kubernetes config=kubernetes-multicontainers-per-pod-with-actuator
time=2025-03-25T08:48:36.015Z level=INFO source=kubernetes.go:323 msg="Using pod service account via in-cluster config" component="discovery manager scrape" discovery=kubernetes config=kubernetes-service-endpoints
time=2025-03-25T08:48:36.016Z level=INFO source=kubernetes.go:323 msg="Using pod service account via in-cluster config" component="discovery manager notify" discovery=kubernetes config=config-0
time=2025-03-25T08:48:36.023Z level=INFO source=main.go:1476 msg="updated GOGC" old=100 new=75
time=2025-03-25T08:48:36.023Z level=INFO source=main.go:1486 msg="Completed loading of configuration file" db_storage=1.009µs remote_storage=1.198µs web_handler=439ns query_engine=1.85µs scrape=311.856µs scrape_sd=1.381318ms notify=591.854µs notify_sd=425.051µs rules=7.281968ms tracing=3.863µs filename=/etc/config/prometheus.yml totalDuration=11.578995ms
time=2025-03-25T08:48:36.023Z level=INFO source=main.go:1213 msg="Server is ready to receive web requests."
time=2025-03-25T08:48:36.023Z level=INFO source=manager.go:175 msg="Starting rule manager..." component="rule manager"
fatal error: sync: unlock of unlocked mutex

goroutine 1306 [running]:
sync.fatal({0x3ef8762?, 0x3594ba0?})
	/usr/local/go/src/runtime/panic.go:1031 +0x18
sync.(*Mutex).unlockSlow(0xc006810e88, 0xffffffff)
	/usr/local/go/src/sync/mutex.go:231 +0x35
sync.(*Mutex).Unlock(...)
	/usr/local/go/src/sync/mutex.go:225
github.com/prometheus/prometheus/tsdb.(*headAppender).AppendHistogramCTZeroSample(0xc006383680, 0x0?, {{0xc00680f680?, 0xc005a97318?}}, 0xbbd1df?, 0x195c929f602, 0xc0067cb4a0, 0x0)
	/app/tsdb/head_append.go:818 +0x5d5
github.com/prometheus/prometheus/storage.(*fanoutAppender).AppendHistogramCTZeroSample(0xc006754270, 0xef45eac?, {{0xc00680f680?, 0x0?}}, 0x195cc7d5fbe, 0x195c929f602, 0xc0067cb4a0, 0x0)
	/app/storage/fanout.go:203 +0x5a
github.com/prometheus/prometheus/scrape.(*scrapeLoop).append(0xc0063034a0, {0x4879000, 0xc006754270}, {0xc006804000?, 0xc006657b58?, 0x415925?}, {0xc0067e0200, 0x72}, {0x1b1accea, 0xedf7464eb, ...})
	/app/scrape/scrape.go:1753 +0xecc
github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport(0xc0063034a0, {0xedf7464eb?, 0x6a10d60?, 0x0?}, {0xedf7464eb?, 0x6a10d60?, 0x6a10d60?}, 0x0)
	/app/scrape/scrape.go:1461 +0xabd
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc0063034a0, 0x0)
	/app/scrape/scrape.go:1351 +0x371
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync in goroutine 1302
	/app/scrape/scrape.go:597 +0xcdd

goroutine 1 [chan receive]:
github.com/oklog/run.(*Group).Run(0xc002a8b180)
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:43 +0x145
main.main.func34(0xc00006e100?, 0xc000665ea0)
	/app/cmd/prometheus/main.go:1370 +0x1f
main.main()
	/app/cmd/prometheus/main.go:1374 +0xaf06

goroutine 18 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x6a11700)
	/go/pkg/mod/k8s.io/klog@v1.0.0/klog.go:1010 +0x66
created by k8s.io/klog.init.0 in goroutine 1
	/go/pkg/mod/k8s.io/klog@v1.0.0/klog.go:411 +0xc8

goroutine 439 [semacquire]:
sync.runtime_Semacquire(0xc000acf338?)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0x3a9ed80?)
	/usr/local/go/src/sync/waitgroup.go:118 +0x48
golang.org/x/sync/errgroup.(*Group).Wait(0xc000ade140)
	/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:56 +0x25
github.com/prometheus/exporter-toolkit/web.ServeMultiple({0xc000a0ae40, 0x1, 0xc000adcb60?}, 0xc000a9c1e0, 0xc000acf320, 0xc000901230)
	/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.13.2/web/tls_config.go:277 +0x117
github.com/prometheus/prometheus/web.(*Handler).Run.func2()
	/app/web/web.go:709 +0xa5
created by github.com/prometheus/prometheus/web.(*Handler).Run in goroutine 307
	/app/web/web.go:708 +0x9d3

goroutine 87 [select]:
github.com/prometheus/prometheus/util/logging.(*Deduper).run(0xc0007da600)
	/app/util/logging/dedupe.go:122 +0xe5
created by github.com/prometheus/prometheus/util/logging.Dedupe in goroutine 1
	/app/util/logging/dedupe.go:49 +0x10a

goroutine 95 [select]:
github.com/prometheus/prometheus/storage/remote.(*WriteStorage).run(0xc000be6dc0)
	/app/storage/remote/write.go:119 +0xbe
created by github.com/prometheus/prometheus/storage/remote.NewWriteStorage in goroutine 1
	/app/storage/remote/write.go:111 +0x52f

goroutine 27 [syscall]:
os/signal.signal_recv()
	/usr/local/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
	/usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 298 [select]:
main.main.func9()
	/app/cmd/prometheus/main.go:1013 +0xac
github.com/oklog/run.(*Group).Run.func1({0xc000a88000?, 0xc0005de000?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x5a

goroutine 299 [chan receive]:
github.com/prometheus/prometheus/discovery.(*Manager).Run(0xc000720a80)
	/app/discovery/manager.go:192 +0x70
main.main.func11()
	/app/cmd/prometheus/main.go:1035 +0x25
github.com/oklog/run.(*Group).Run.func1({0xc000a8c000?, 0xc000a8c018?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x5a

goroutine 300 [chan receive]:
github.com/prometheus/prometheus/discovery.(*Manager).Run(0xc000720b40)
	/app/discovery/manager.go:192 +0x70
main.main.func13()
	/app/cmd/prometheus/main.go:1049 +0x25
github.com/oklog/run.(*Group).Run.func1({0xc000a8c030?, 0xc000a8c048?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x5a

goroutine 301 [chan receive]:
github.com/prometheus/prometheus/rules.(*Manager).Run(...)
	/app/rules/manager.go:177
main.main.func15()
	/app/cmd/prometheus/main.go:1064 +0x79
github.com/oklog/run.(*Group).Run.func1({0xc000a8c060?, 0xc000a8a010?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x5a

goroutine 302 [select]:
github.com/prometheus/prometheus/scrape.(*Manager).Run(0xc000be1c80, 0xc000118bd0)
	/app/scrape/manager.go:124 +0xd1
main.main.func17()
	/app/cmd/prometheus/main.go:1082 +0x51
github.com/oklog/run.(*Group).Run.func1({0xc000a88030?, 0xc000a8c078?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x5a

goroutine 303 [chan receive]:
github.com/prometheus/prometheus/tracing.(*Manager).Run(0xc000017680)
	/app/tracing/tracing.go:66 +0xd5
main.main.func19()
	/app/cmd/prometheus/main.go:1101 +0x2f
github.com/oklog/run.(*Group).Run.func1({0xc000a8c090?, 0xc000a8a020?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x5a

goroutine 304 [select]:
main.main.func22()
	/app/cmd/prometheus/main.go:1139 +0x175
github.com/oklog/run.(*Group).Run.func1({0xc000778a80?, 0xc000a0ae80?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x5a

goroutine 305 [chan receive]:
main.main.func24()
	/app/cmd/prometheus/main.go:1214 +0x235
github.com/oklog/run.(*Group).Run.func1({0xc00029b740?, 0xc000a0ae90?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x5a

goroutine 306 [chan receive]:
main.main.func26()
	/app/cmd/prometheus/main.go:1267 +0x6ec
github.com/oklog/run.(*Group).Run.func1({0xc0005c3450?, 0xc000a0f290?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x5a

goroutine 307 [select]:
github.com/prometheus/prometheus/web.(*Handler).Run(0xc000777540, {0x485ded0, 0xc0009bb9a0}, {0xc000a0ae40, 0x1, 0x1}, {0x0, 0x0})
	/app/web/web.go:712 +0xa4d
main.main.func30()
	/app/cmd/prometheus/main.go:1337 +0x36
github.com/oklog/run.(*Group).Run.func1({0xc000a10880?, 0xc000a0aea0?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x5a

goroutine 308 [semacquire]:
sync.runtime_Semacquire(0xc000be42b8?)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc00029b7a0?)
	/usr/local/go/src/sync/waitgroup.go:118 +0x48
github.com/prometheus/prometheus/notifier.(*Manager).Run(0xc000666e70, 0xc000118d20)
	/app/notifier/notifier.go:344 +0xf1
main.main.func32()
	/app/cmd/prometheus/main.go:1360 +0x51
github.com/oklog/run.(*Group).Run.func1({0xc000a0f2c0?, 0xc000a0aeb0?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x5a

goroutine 309 [select]:
github.com/prometheus/prometheus/discovery.(*Manager).sender(0xc000720a80)
	/app/discovery/manager.go:356 +0xc8
created by github.com/prometheus/prometheus/discovery.(*Manager).Run in goroutine 299
	/app/discovery/manager.go:191 +0x56

goroutine 310 [select]:
github.com/prometheus/prometheus/discovery.(*Manager).sender(0xc000720b40)
	/app/discovery/manager.go:356 +0xc8
created by github.com/prometheus/prometheus/discovery.(*Manager).Run in goroutine 300
	/app/discovery/manager.go:191 +0x56

goroutine 440 [IO wait]:
internal/poll.runtime_pollWait(0x7fdacaad3e50, 0x72)
	/usr/local/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc00090a500?, 0x415925?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00090a500)
	/usr/local/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc00090a500)
	/usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc000a10740)
	/usr/local/go/src/net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc000a10740)
	/usr/local/go/src/net/tcpsock.go:372 +0x30
github.com/prometheus/prometheus/util/netconnlimit.(*sharedLimitListener).Accept(0xc000a0f1a0)
	/app/util/netconnlimit/netconnlimit.go:73 +0x85
github.com/mwitkow/go-conntrack.(*connTrackListener).Accept(0xc000831cb0)
	/go/pkg/mod/github.com/mwitkow/go-conntrack@v0.0.0-20190716064945-2f068394615f/listener_wrapper.go:100 +0x5c
net/http.(*Server).Serve(0xc000a9c1e0, {0x48464e0, 0xc000831cb0})
	/usr/local/go/src/net/http/server.go:3330 +0x30c
github.com/prometheus/exporter-toolkit/web.Serve({0x48464e0, 0xc000831cb0}, 0xc000a9c1e0, 0xc000acf320, 0xc000901230)
	/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.13.2/web/tls_config.go:351 +0x756
github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1()
	/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.13.2/web/tls_config.go:274 +0x27
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 439
	/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96

goroutine 465 [select]:
github.com/prometheus/prometheus/tsdb/wlog.(*WL).run(0xc000af8b00)
	/app/tsdb/wlog/wlog.go:416 +0x96
created by github.com/prometheus/prometheus/tsdb/wlog.NewSize in goroutine 306
	/app/tsdb/wlog/wlog.go:375 +0x5f8

goroutine 1265 [select]:
github.com/prometheus/prometheus/rules.(*Group).run(0xc005d58fc0, {0x485de28, 0x6a35580})
	/app/rules/group.go:214 +0x15f
github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0xc005d58fc0)
	/app/rules/manager.go:258 +0xa8
created by github.com/prometheus/prometheus/rules.(*Manager).Update in goroutine 305
	/app/rules/manager.go:248 +0x3b6

goroutine 1184 [select]:
github.com/prometheus/prometheus/rules.(*Group).run(0xc005d41e60, {0x485de28, 0x6a35580})
	/app/rules/group.go:214 +0x15f
github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0xc005d41e60)
	/app/rules/manager.go:258 +0xa8
created by github.com/prometheus/prometheus/rules.(*Manager).Update in goroutine 305
	/app/rules/manager.go:248 +0x3b6

goroutine 1183 [select]:
github.com/prometheus/prometheus/rules.(*Group).run(0xc005d41b00, {0x485de28, 0x6a35580})
	/app/rules/group.go:262 +0x466
github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0xc005d41b00)
	/app/rules/manager.go:258 +0xa8
created by github.com/prometheus/prometheus/rules.(*Manager).Update in goroutine 305
	/app/rules/manager.go:248 +0x3b6

goroutine 1182 [select]:
github.com/prometheus/prometheus/rules.(*Group).run(0xc005d41320, {0x485de28, 0x6a35580})
	/app/rules/group.go:214 +0x15f
github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0xc005d41320)
	/app/rules/manager.go:258 +0xa8
created by github.com/prometheus/prometheus/rules.(*Manager).Update in goroutine 305
	/app/rules/manager.go:248 +0x3b6

goroutine 1181 [select]:
github.com/prometheus/prometheus/discovery.(*Manager).updater(0xc000720b40, {0x485ded0, 0xc0008367d0}, 0xc0002dad20, 0xc00584c460)
	/app/discovery/manager.go:325 +0x12e
created by github.com/prometheus/prometheus/discovery.(*Manager).startProvider in goroutine 305
	/app/discovery/manager.go:304 +0x31f

goroutine 1180 [semacquire]:
sync.runtime_Semacquire(0xc0057350a0?)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc000756780?)
	/usr/local/go/src/sync/waitgroup.go:118 +0x48
github.com/prometheus/prometheus/discovery/kubernetes.(*Discovery).Run(0xc000756780, {0x485ded0, 0xc0008367d0}, 0xc00584c460)
	/app/discovery/kubernetes/kubernetes.go:615 +0x727
created by github.com/prometheus/prometheus/discovery.(*Manager).startProvider in goroutine 305
	/app/discovery/manager.go:303 +0x292

goroutine 1191 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0068224f8, 0x1)
	/usr/local/go/src/runtime/sema.go:587 +0x159
sync.(*Cond).Wait(0xc005d79ea0?)
	/usr/local/go/src/sync/cond.go:71 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0068224d0, 0xc000a0b200)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/delta_fifo.go:588 +0x231
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc006822580)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/controller.go:195 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00574fe60, {0x48289a0, 0xc00573acf0}, 0x1, 0xc005738af0)
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00574fe60, 0x3b9aca00, 0x0, 0x1, 0xc005738af0)
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc006822580, 0xc005738af0)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/controller.go:166 +0x375
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0068222c0, 0xc005738af0)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/shared_informer.go:508 +0x2a9
created by github.com/prometheus/prometheus/discovery/kubernetes.(*Discovery).Run in goroutine 1178
	/app/discovery/kubernetes/kubernetes.go:513 +0x1a1f

goroutine 1173 [chan receive]:
github.com/prometheus/prometheus/discovery.(*Manager).updater(0xc000720a80, {0x485ded0, 0xc000b050e0}, 0xc0002c7d50, 0xc00584c2a0)
	/app/discovery/manager.go:333 +0x2e6
created by github.com/prometheus/prometheus/discovery.(*Manager).startProvider in goroutine 305
	/app/discovery/manager.go:304 +0x31f

goroutine 1192 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc006822708, 0x1)
	/usr/local/go/src/runtime/sema.go:587 +0x159
sync.(*Cond).Wait(0xc005e2e480?)
	/usr/local/go/src/sync/cond.go:71 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0068226e0, 0xc000a0b650)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/delta_fifo.go:588 +0x231
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc006822790)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/controller.go:195 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00572de60, {0x48289a0, 0xc00573b140}, 0x1, 0xc005738af0)
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00572de60, 0x3b9aca00, 0x0, 0x1, 0xc005738af0)
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc006822790, 0xc005738af0)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/controller.go:166 +0x375
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc006822370, 0xc005738af0)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/shared_informer.go:508 +0x2a9
created by github.com/prometheus/prometheus/discovery/kubernetes.(*Discovery).Run in goroutine 1178
	/app/discovery/kubernetes/kubernetes.go:514 +0x1ac9

goroutine 1296 [select]:
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc0064d51e0, 0x0)
	/app/scrape/scrape.go:1305 +0xde
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync in goroutine 1301
	/app/scrape/scrape.go:597 +0xcdd

goroutine 1169 [chan receive]:
github.com/prometheus/prometheus/discovery.(*Manager).updater(0xc000720a80, {0x485ded0, 0xc000b05040}, 0xc0002c7c70, 0xc00584c1c0)
	/app/discovery/manager.go:333 +0x2e6
created by github.com/prometheus/prometheus/discovery.(*Manager).startProvider in goroutine 305
	/app/discovery/manager.go:304 +0x31f

goroutine 1194 [chan receive]:
github.com/prometheus/prometheus/discovery/kubernetes.(*Endpoints).Run(0xc005e48d80, {0x485ded0, 0xc000b05270}, 0xc00584c3f0)
	/app/discovery/kubernetes/endpoints.go:237 +0x374
github.com/prometheus/prometheus/discovery/kubernetes.(*Discovery).Run.func19({0x481f160?, 0xc005e48d80?})
	/app/discovery/kubernetes/kubernetes.go:609 +0x62
created by github.com/prometheus/prometheus/discovery/kubernetes.(*Discovery).Run in goroutine 1178
	/app/discovery/kubernetes/kubernetes.go:607 +0x5db

goroutine 1266 [select]:
github.com/prometheus/prometheus/rules.(*Group).run(0xc005db2900, {0x485de28, 0x6a35580})
	/app/rules/group.go:214 +0x15f
github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0xc005db2900)
	/app/rules/manager.go:258 +0xa8
created by github.com/prometheus/prometheus/rules.(*Manager).Update in goroutine 305
	/app/rules/manager.go:248 +0x3b6

goroutine 1267 [select]:
github.com/prometheus/prometheus/notifier.(*Manager).targetUpdateLoop(0xc000666e70, 0xc000118d20)
	/app/notifier/notifier.go:381 +0x99
github.com/prometheus/prometheus/notifier.(*Manager).Run.func1()
	/app/notifier/notifier.go:335 +0x4b
created by github.com/prometheus/prometheus/notifier.(*Manager).Run in goroutine 308
	/app/notifier/notifier.go:333 +0x99

goroutine 1261 [select]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc0009b41e0, 0xc000bedea0, 0xc005a8fb20, 0xc0005fc700)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/reflector.go:381 +0x106
created by k8s.io/client-go/tools/cache.(*Reflector).watchWithResync in goroutine 1168
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/reflector.go:405 +0x105

goroutine 1168 [select]:
k8s.io/client-go/tools/cache.handleAnyWatch({0xc006058460?, 0xa?, 0x6a10d60?}, {0x483dd98, 0xc0061fd7c0}, {0x7fda76becce8, 0xc0068220b0}, {0x48a6eb0, 0x3e523c0}, 0x0, ...)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/reflector.go:794 +0x230
k8s.io/client-go/tools/cache.handleWatch({0xa?, 0x0?, 0x6a10d60?}, {0x483dd98?, 0xc0061fd7c0?}, {0x7fda76becce8?, 0xc0068220b0?}, {0x48a6eb0?, 0x3e523c0?}, 0x0, ...)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/reflector.go:760 +0xdd
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc0009b41e0, {0x0?, 0x0?}, 0xc000bedea0, 0xc0005fc700)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/reflector.go:458 +0x556
k8s.io/client-go/tools/cache.(*Reflector).watchWithResync(0xc0009b41e0, {0x0, 0x0}, 0xc000bedea0)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/reflector.go:406 +0x12c
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0009b41e0, 0xc000bedea0)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/reflector.go:369 +0x3c5
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/reflector.go:307 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc005743f50, {0x48289c0, 0xc000b98230}, 0x1, 0xc000bedea0)
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0009b41e0, 0xc000bedea0)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/reflector.go:306 +0x1af
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/wait.go:72 +0x4c
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 1162
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/wait.go:70 +0x73

goroutine 1167 [chan receive]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/controller.go:138 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 1162
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/controller.go:137 +0xab

goroutine 1268 [select]:
github.com/prometheus/prometheus/notifier.(*Manager).sendLoop(0xc000666e70)
	/app/notifier/notifier.go:357 +0x72
github.com/prometheus/prometheus/notifier.(*Manager).Run.func2()
	/app/notifier/notifier.go:340 +0x4c
created by github.com/prometheus/prometheus/notifier.(*Manager).Run in goroutine 308
	/app/notifier/notifier.go:338 +0xe7

goroutine 1166 [chan receive]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc000b980f0, 0xc0057380e0)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/shared_informer.go:807 +0x4d
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/wait.go:72 +0x4c
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 1162
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/wait.go:70 +0x73

goroutine 1177 [chan receive]:
github.com/prometheus/prometheus/discovery.(*Manager).updater(0xc000720a80, {0x485ded0, 0xc000b05220}, 0xc0002c7e30, 0xc00584c380)
	/app/discovery/manager.go:333 +0x2e6
created by github.com/prometheus/prometheus/discovery.(*Manager).startProvider in goroutine 305
	/app/discovery/manager.go:304 +0x31f

goroutine 1175 [chan receive]:
github.com/prometheus/prometheus/discovery.(*Manager).updater(0xc000720a80, {0x485ded0, 0xc000b05130}, 0xc0002c7dc0, 0xc00584c310)
	/app/discovery/manager.go:333 +0x2e6
created by github.com/prometheus/prometheus/discovery.(*Manager).startProvider in goroutine 305
	/app/discovery/manager.go:304 +0x31f

goroutine 1190 [chan receive]:
k8s.io/client-go/util/workqueue.(*Typed[...]).updateUnfinishedWorkLoop(0x4883760)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/util/workqueue/queue.go:360 +0x97
created by k8s.io/client-go/util/workqueue.newQueue[...] in goroutine 1178
	/go/pkg/mod/k8s.io/client-go@v0.31.3/util/workqueue/queue.go:180 +0x1e9

goroutine 1171 [chan receive]:
github.com/prometheus/prometheus/discovery.(*Manager).updater(0xc000720a80, {0x485ded0, 0xc000b05090}, 0xc0002c7ce0, 0xc00584c230)
	/app/discovery/manager.go:333 +0x2e6
created by github.com/prometheus/prometheus/discovery.(*Manager).startProvider in goroutine 305
	/app/discovery/manager.go:304 +0x31f

goroutine 1193 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc006822918, 0x2)
	/usr/local/go/src/runtime/sema.go:587 +0x159
sync.(*Cond).Wait(0xc0063247c0?)
	/usr/local/go/src/sync/cond.go:71 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0068228f0, 0xc000a0bb30)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/delta_fifo.go:588 +0x231
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0068229a0)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/controller.go:195 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00574be60, {0x48289a0, 0xc00573b560}, 0x1, 0xc005738af0)
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00574be60, 0x3b9aca00, 0x0, 0x1, 0xc005738af0)
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc0068229a0, 0xc005738af0)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/controller.go:166 +0x375
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc006822420, 0xc005738af0)
	/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/shared_informer.go:508 +0x2a9
created by github.com/prometheus/prometheus/discovery/kubernetes.(*Discovery).Run in goroutine 1178
	/app/discovery/kubernetes/kubernetes.go:515 +0x1187

goroutine 1322 [IO wait]:
internal/poll.runtime_pollWait(0x7fdacaad39f0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc00665d000?, 0xc0067a3000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00665d000, {0xc0067a3000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc00665d000, {0xc0067a3000?, 0x4b35c5?, 0x485df40?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc000bac040, {0xc0067a3000?, 0xc0064d9f80?, 0xc006750290?})
	/usr/local/go/src/net/net.go:189 +0x45
net/http.(*persistConn).Read(0xc00674a5a0, {0xc0067a3000?, 0x77a3e5?, 0x35d6d80?})
	/usr/local/go/src/net/http/transport.go:2052 +0x4a
bufio.(*Reader).fill(0xc00669f5c0)
	/usr/local/go/src/bufio/bufio.go:110 +0x103
bufio.(*Reader).Peek(0xc00669f5c0, 0x1)
	/usr/local/go/src/bufio/bufio.go:148 +0x53
net/http.(*persistConn).readLoop(0xc00674a5a0)
	/usr/local/go/src/net/http/transport.go:2205 +0x185
created by net/http.(*Transport).dialConn in goroutine 1331
	/usr/local/go/src/net/http/transport.go:1874 +0x154f
...

Metadata
Metadata
Assignees
Labels
kind/bug
Type
No type
Projects
No projects
Milestone
No milestone
Relationships
None yet
Development
No branches or pull requests
Issue actions