Nice! I believe the only question is -- do we (and users) accept the kind of errors you will get from calculating this from fixed buckets e.g. if you have only le=0.1 and le=10, le=100 and you do histogram_fraction(1, 15, rate(http_request_duration_seconds_bucket[1h])) -- the errors in this case will make the result pretty useless (no?). We don't have a good way to share pointers on what start and end time makes sense given buckets. Also is using _bucket series makes sense here? Will we use sum/count?

Anyway, I think it makes sense to extrapolate some usefulness here anyway, with the right documentations and maybe errors if potential start,end does not make sense? 🤔 I'm sure @beorn7 has opinions on this already too.

Nice! I believe the only question is -- do we (and users) accept the kind of errors you will get from calculating this from fixed buckets e.g. if you have only le=0.1 and le=10, le=100 and you do histogram_fraction(1, 15, rate(http_request_duration_seconds_bucket[1h])) -- the errors in this case will make the result pretty useless (no?). We don't have a good way to share pointers on what start and end time makes sense given buckets. Also is using _bucket series makes sense here? Will we use sum/count?

Anyway, I think it makes sense to extrapolate some usefulness here anyway, with the right documentations and maybe errors if potential start,end does not make sense? 🤔 I'm sure @beorn7 has opinions on this already too.

We have to use the _buckets to get the "le" label at least. The calculation is essentially the same as for native histograms, it just depends on "good" choices for bucket boundaries more ( i wonder if linear interpolation is right choice though! ). If we agree that such a function would be valuable then we should add the documentation that its an approximation if the "upper/lower" parameters do not align with any bucket boundaries though - i think that is sensible!

Nitty technical thoughts and answers:

• Linear interpolation makes most sense, mostly for consistency, because we have used linear interpolation for quantile estimation with classic histograms forever. We have even decided to keep it that way for NHCB. Given how users can set all kind of bucket layouts, it's hard to to something that is generally useful, so we could as well just use the simplest method, which is at least easy to reason with. (A maybe a bit better rationale is that one of the remaining use cases to prefer NHCB over standard exponential buckets are distributions that don't map nicely to the exponential buckets, and those might often be more-or-less linear distributions – like stuff that is already logarithmic (dB, star brightness, …) or "artificial" units that are designed to approach linear distributions ("scores" or "grades" of any kind: SpamAssassin, school grades, star ratings).
• It would complicate things a lot if we needed the count and sum series (because it is a completely separate series with classic histograms). Luckily, we have the +Inf bucket, which is by definition the same as the count (note that this is different for native histograms because NaN observations go into no bucket, but that's just a weird edge case). This also means that we require the +Inf bucket. The implementation in this PR already handles this correctly (be returning NaN if it isn't there), but obviously, this needs to be documented explicitly in the end.

My general thoughts:

tl;dr: If a sufficient amount of people desire this sufficiently strongly, I guess we can add this. There is not much technical downside to adding it (implementation is not very complicated; it doesn't add "yet another function" to PromQL, just modifies the behavior of an existing function). I'm not convinced, though, that this is something that people should desire…

I haven't really bought yet the two prime incentives for this:

1. "easing the transition to native histograms later on": You still have to change the query when moving from classic histograms to native histograms. Yes, it is a smaller change. But you also have to change from whatever you have now to histogram_fraction, after merging this PR. So why not migrate to native histograms (could be NHCB) normally and change the query only once?
2. "without knowing the actual values of the 'le' label": I see this more as a problem than a benefit. Given that classic histograms are usually very low-res and in particular only cover a limited range of observed value, not knowing what your le values are is a recipe for disaster. (This echos @bwplotka's concern above.)

histogram_fraction was introduced for native histograms because there is no way of accessing individual buckets, and because of the dynamic nature of the buckets. (Or in other words: The reason was not that histogram_fraction was considered a better interface than the direct access of buckets possible with classic histograms. More the contrary…)

Conclusion: If you left it to me to decide, I would say no. But whoever feels strongly that we should have this, please say so. You don't even have to convince me. You just have to state that you are convinced. If there is critical mass for this (and no objections against it otherwise), I will not oppose it.

So why not migrate to native histograms (could be NHCB) normally and change the query only once?

The problem is this work is not "instant".

We have thousands of Prometheus instances for thousands of services. It's going to take us months to migrate and cross-coordinate tooling with teams. And we might need to rollout and rollback native histograms.

So by having the option to pre-convert queries from classic to native allows us to stagger the changes across various teams.

So, I feel fairly strongly that this is something very useful for us to support. It allows for flexibility forwards and backwards in mixed deployment environments.

So by having the option to pre-convert queries from classic to native allows us to stagger the changes across various teams.

But the query is still different, isn't it? Or are you doing something like histogram_fraction(1, 15, rate({__name__=~"http_request_duration_seconds.*"}[1h])) ?

Straw poll on Slack got a few thumbs up and no thumbs down. So let's do this. I'll review this PR ASAP.

@MichaHoffmann could you update the documentation in the meantime? It should be part of this PR.

Generally looks good. A bit more testing would be good. Also to reduce the code duplication where it is easy. (But don't sweat it. I think a bit of duplication is better than convoluted contraptions to make the code as DRY as possible.)

And as said before, we need an update for the documentation.

Thank you! Ill address the comments over the weekend!

@MichaHoffmann are you still working on this? As said, we are only missing some documentation improvements.

@MichaHoffmann are you still working on this? As said, we are only missing some documentation improvements.

Yeah! Will add your suggestions - I just didn't find a lot of energy and time lately to do it ! Ill do it over the weekend

Thanks.

What do you think about my refinement?

Yeah, reads good to me, i committed the suggestion!