@dgl given you're both the promtool maintainer and suggested this approach in #5352 (comment)...

I like the concept. Maybe rather than a command line flag it should be configured in the test file. Putting it in the test file makes it more explicit, makes the test self contained and avoids a whole "works on my machine" and "how are you running the tests?" situation with the obvious downside this then becomes a piece of boilerplate.

Another reason to configure this in the file is while I've only ever seen off-by-one value errors we could also allow a file to override with off-by-two, off-by-x if anyone sees such an issue... (I am not suggesting to implement that initially but it leaves the option open to have this per test file.)

The "works on my machine" argument is compelling enough, I'll re-work with a fuzzy_match file scope bool (barring any better naming suggestions).

In the promql testing framework, we have been using almost.Equal for a while. That's even more tolerant than the approach here, but maybe that's fine?

Well now I'm tempted to just expose the epsilon of almost.Equal as a configurable and then we never have to worry if people want seriously weak comparisons.

I think I'd rather stick with the "off-by-one" for now as that's also the only case I've run into in my rule tests. This PR should provide an obvious path to adding another option should that be necessary.

In the promql testing framework, we have been using almost.Equal for a while. That's even more tolerant than the approach here, but maybe that's fine?

Given it's more tolerant we could change to it later if we liked. Lets go with this.