/prombench v3.0.1

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-15683 and v3.0.1

After the successful deployment (check status here), the benchmarking results can be viewed at:

• Prometheus Meta
• Prombench Dashboard
• Grafana Explorer, Loki logs
• Parca profiles (e.g. in-use memory)

Available Commands:

• To restart benchmark: /prombench restart v3.0.1
• To stop benchmark: /prombench cancel
• To print help: /prombench help

Some learnings so far @dashpole looking on http://prombench.prometheus.io/grafana/d/7gmLoNDmz/prombench?orgId=1&var-pr-number=15683&var-RuleGroup=All&var-prNumber=15683&from=1734426543125&to=1734512943126

Somehow this change either reduces the cardinality of Prometheus OR stops Prometheus from ingesting all data 🤔

I see 3.0 have 2-4x times more metrics and chunks. Our source of metrics as of now (we are switching to avalanche soon): (config, deploy, code).

37 times more allocations for this PR, which suggest something is off.

This PR uses less memory overall, likely because of less metrics being stored.

This PR uses 5x more CPU

We kind of scrape the same amount of samples though:

Nothing super interesting in logs other than occasional failed scrape targets.

Scrapes look healthy: http://34.89.207.251/15683/prometheus-pr/targets?pool=

37 times more allocations for this PR, which suggest something is off.

Comparing profiles shows problems in query/query_range APIs

/prombench cancel

Going to cancel prombench run, so we can unblock other benchmarks runs.

Benchmark cancel is in progress.

As agreed in our sync I rebased and only add label in a separate PR #16025 -- I noticed potential alternative when it comes to storage model.

I'm closing this one since #16228 got merged, please let me know if it should remain open!