Proposal

Right now, Prometheus supports scraping metrics either uncompressed or via gzip.

I created arbitrary metrics via avalanche:

docker run -p 9001:9001 quay.io/prometheuscommunity/avalanche:main  --metric-count=50000
curl localhost:9001 > foo.metrics

When compressing these via the command line, it looks like zstd has a bit higher CPU usage, but a better Wall time.

time gzip -k foo.metrics
gzip -k foo.metrics  0,97s user 0,03s system 99% cpu 1,000 total
time zstd -k foo.metrics
zstd -k foo.metrics  0,13s user 0,05s system 151% cpu 0,113 total

At the same time, compression for metrics seems make them 27.5% smaller than with gzip:

From this perspective, it might be promising to support additional zstd content-encoding for Prometheus Scrapes as well, so Prometheus could offer "zstd, gzip" as options (and fall back on gzip or even uncompressed) if the scrape target does not support it.

A possible candidate for this could be kube-state-metrics, which is creates a lot of metrics with lots of labels that could compress well with zstd.

Unrelated to the prometheus ecosystem, it might be interesting to see if some http based benchmarks appear, Chrome will support content-encoding with zstd as well: https://chromestatus.com/feature/6186023867908096