One example where we certainly didn't support all intermediate versions: From NixOS 24.11 on, we bumped minver to 2.3.17 to support zstd compressed binary artifacts. That doesn't mean that Nix 2.4, 2.5, ... etc. all support these, though. So we were essentially already breaking the minver promise for a long time.

AFAIK zstd support was introduced when nix moved to libarchive, which was in 2.4.

AFAIK zstd support was introduced when nix moved to libarchive, which was in 2.4.

Oh, OK. Then it's very confusing why the bump for related minver appears in the release notes for NixOS 24.11, which is well beyond the release of Nix 2.4, right? Maybe it was only backported to 2.3 that late.

I'll scratch that example, although the general argument probably is still valid with so many intermediate versions not being tested explicitly.

Yes, Zstandard was backported late. I believe there are other backported things in 2.3 that aren’t in 2.4, though, like structured attributes stuff.

(The cache doesn’t actually use Zstandard yet, so I think no practical compat break even though a declared one. It’s worth noting that the versions of NixOS old enough to ship with 2.3 don’t support it, though, so in terms of “long eval‐compatibility timeline” it’s effectively a 2.4 requirement.)

FWIW my personal feeling is that moving on this probably only makes sense once we actually have a new feature to add to the list and… I’m not yet sure what that would take or what the feature would be. (We will have to do something like this at some point, of course.)

FWIW my personal feeling is that moving on this probably only makes sense once we actually have a new feature to add to the list

I think moving on it now makes sense, because it clearly clarifies out intent regarding "Nix 2.18". Is that actually supported or not?

I'd argue that we don't support Nix 2.18, because we can't easily test it. And because it's insecure and unmaintained. But we can't bump to minver 2.24, because Lix exposes a different version number. Thus - the feature-check is already required.

I think I’d at least merge a PR that fixes an eval issue on Nix 2.18, currently. Supporting upgrades from 23.11 seems like a good thing – you could have a machine that’s in support in June 2024, put it in storage for a year, and then pick it back up now and want to upgrade to unstable. It’s not the end of the world to be expected to go through an intermediate Nix but I also think it’s not unreasonable to expect it to still work.

Note that we always have incomplete coverage for everything but the versions running on Hydra, because eval is only a small part of the puzzle, and even a vanilla NixOS system requires running builds. And we also don’t have full eval coverage for Lix now, either, even if there are fewer obstacles to fixing that than with 2.18. So while the things we can test automatically will always have more first‐class support than things we don’t, I think some level of second‐tier “supported, but in a reactive way” is somewhat inevitable given the number of different configurations you can use Nixpkgs with. (Example: Case‐sensitive stores on macOS – which should become the default once the preparatory work to enable that is done, but there will certainly be a time period where both configurations coexist.)

And we also don’t have full eval coverage for Lix now, either, even if there are fewer obstacles to fixing that than with 2.18.

It depends what you mean by "full". We do Eval with all Lix versions now, just not on every PR.

I think I’d at least merge a PR that fixes an eval issue on Nix 2.18, currently. Supporting upgrades from 23.11 seems like a good thing

Note that this level of support is still possible with the new features-based approach. We just don't commit to "full support" for Nix 2.18 anymore. It might work, it might not.

It's orthogonal to this PR. No matter whether we do "min ver" or "features", we always have:

• 100% support as in: we build with this version on hydra
• "full" support as in: we do Eval Nixpkgs with this version regularly and commit to fixing regressions.
• "some" support as in: we're not testing it via Eval, but we're also not blocking evaluation of Nixpkgs with it in default.nix.
• no support as in: we're knowingly using features that this version doesn't support and are blocking it in default.nix.

Traditionally, the "some support" category has been a lot bigger. We have recently cut it down from both sides:

• raised minver from 2.3 to 2.18
• started regular Eval with all Lix and Nix versions available in Nixpkgs.

While "some" is a lot smaller now, it will always remain. Currently it includes versions like Nix 2.18, 2.19, 2.20, 2.21, 2.22, 2.23, 2.25, 2.26, 2.27 and Lix 2.90, I think.

This is not going away. I am hoping, though, that with the approach in this PR we are moving towards a better communication of these facts. We are not simply communicating "You need Nix >= 2.18" anymore, which might be misleading as in "We are sure Nix 2.18 will work".

(although yes, I agree, the difference between "full" and "some" is not clearly communicated anywhere, still)

(although yes, I agree, the difference between "full" and "some" is not clearly communicated anywhere, still)

I briefly wished to be able to automatically list all "fully supported" nix versions in the error message in default.nix, aka those available in the current Nixpkgs. But that doesn't work... I'd need to evaluate Nixpkgs to even get to the list :/

Hmm. My personal feeling is that this is orthogonal to communication around “reactive support”, because “not having an eval semantics corner case change that was introduced in 2.19 and unintentionally reverted in 2.22 and we didn’t notice until someone opened a bug” isn’t really a “feature” we’d declare. I think that what we’re currently saying is that we do want to fully support everything that is compatible with Nix 2.18 or later versions, which I believe is the correct stance for us to have at this point (while also thinking that it will make sense to lock down that support window further as we figure out how migration will work and how we expect the ecosystem to involve). The fact that we have the strongest validation of that support for 2.29 (of all versions!) is another matter.

In an ideal world we’d have a 1:1 match between things we want to fully support and what we check automatically, down to a full independent Lix‐based Hydra. (Would be great for checking reproducible builds!) But in the meantime I don’t think that changes in declared expectations could close the gap between our declared expectations and our verified expectations.

I do agree that the good outcome looks like having a valuable feature not present in 2.18 that has agreed semantics between Nix and Lix (and Snix/Tvix?), a robust feature detection story, and no feature flag gating, that we then decide to adopt in Nixpkgs, while having a sensible story around how long we want that to be the case before adopting it, how we expect updates from older versions to work, and so on. And then, indeed, we no longer support 2.18. At that point, however, I wonder if we still explicitly check for 2.18 at all? We would have a feature detection mechanism and no version that declares itself to be older than 2.18 would declare support for that feature, so the 2.18 check would be redundant.

I think that what we’re currently saying is that we do want to fully support everything that is compatible with Nix 2.18 or later versions, which I believe is the correct stance for us to have at this point

I think that's what I intend to change. My intent is to explicitly say "we do not even want to fully support Nix 2.18".

I see no value in it. Updating an older install in one step instead of multiple.. seems not convincing enough for me. Not convincing enough for me to have to guarantee it. Chances that it will work are extremely high anyway, because we are testing both Lix and Nix now. The combination of both gives us a reasonable verification for the fork point as well - mistakes, unintended or intended changes would have to happen across two different codebases.

Saying "we want this to be supported", but not doing anything for it seems pointless to me. I think it's enough to test all Lix / Nix versions for Eval like we currently do. And once we commit to not testing Nix 2.18 explicitly, we are also saying, that we don't want to fully support it.

That being said, I think the outcome that I am hoping for from this discussion is more of "We don't intend to explicitly test with Nix 2.18". I don't want to go that way (testing Nix 2.18), I'd like us to commit to the "we test everything that is available in Nixpkgs" approach.

Whether we use features or minver for the sanity check, I don't really care - it just resonated with me a lot when it was pointed out that minver doesn't make much sense conceptually anymore.

I think that's what I intend to change. My intent is to explicitly say "we do not even want to fully support Nix 2.18".

I see no value in it. Updating an older install in one step instead of multiple.. seems not convincing enough for me. Not convincing enough for me to have to guarantee it. Chances that it will work are extremely high anyway, because we are testing both Lix and Nix now. The combination of both gives us a reasonable verification for the fork point as well - mistakes, unintended or intended changes would have to happen across two different codebases.

Well… I think there are quite a lot of eval regressions in those middle Nix versions that were fixed in both Lix and Nix since. I think chances it’ll work are certainly good, though, yes, which is why I think a reactive approach to addressing issues is fine.

I agree that requiring stepwise‐ish upgrades seems fine, but I do think we probably want a release or two’s slack given Hydra deployments building multiple Nixpkgs versions and so on. I also think this is something we’d want to document, and in fact potentially warn about with more version checks, to determine if a user is “skipping a release” in a way we don’t guarantee will work, so as to inform them about the intended upgrade path.

Basically, I think this is a defensible position, but I also think it’s a further change, rather than just documenting that status quo now that we’re declaring 2.18.

Saying "we want this to be supported", but not doing anything for it seems pointless to me. I think it's enough to test all Lix / Nix versions for Eval like we currently do. And once we commit to not testing Nix 2.18 explicitly, we are also saying, that we don't want to fully support it.

This is probably the core of our disagreement, unless we just take different implications from “fully support”. I think it is inevitable that we will say we want to fully support things but that it will not be practical to exhaustively test them, because there are more combinatorial factors than just the version. (I gave case‐sensitive stores as an example on the build end for macOS; for an eval one, there are also differences in regex handling in supported versions, depending on what C++ standard library implementation is used. )

So it seems inevitable to me, at least for now, that we will have configurations we want to support but do not have automatic end‐to‐end testing of the entire system to cover. This is a second tier of practical support insofar as issues will only be fixed reactively rather than proactively, but I think any issues that crop up with such configurations would be still be legitimate bugs we should fix, and treating them as bugs we should fix rather than WONTFIX issues is precisely the thing we’d be “doing for it”. The cache statistics do show a substantial portion of requests from Nix versions between 2.18 and 2.24 – more than requests from Lix users, even.

That being said, I think the outcome that I am hoping for from this discussion is more of "We don't intend to explicitly test with Nix 2.18". I don't want to go that way (testing Nix 2.18), I'd like us to commit to the "we test everything that is available in Nixpkgs" approach.

Yes, I agree with that. It does not seem worth the resources and maintenance effort. The most likely source of issues would be language features added after Nix 2.18, which a Lix check should reasonably cover. (Well, I agree with “not testing things not available in Nixpkgs”, at least; I don’t know if it makes sense to spend a ton of eval resources on checking a bunch of separate Lix versions for every PR in the merge queue, for instance – and anything we don’t do in the merge queue is a second tier of support already.)

Whether we use features or minver for the sanity check, I don't really care - it just resonated with me a lot when it was pointed out that minver doesn't make much sense conceptually anymore.

I agree that we are entering a “post‐minver” era of sorts, in that it seems very unlikely we would ever just declare free reign on everything post‐2.18 Nix decided to add/change at this point. And I agree that feature testing is the way forward. I think, though, that we are in a transition period where it does make sense to treat Nix 2.18+ as legitimately supported, and that moving away from version checks to feature checks entirely will depend on what that features is, how detection of it ends up looking, and perhaps better communication or tooling about how to do stepwise upgrades. And I also think that this would be a further change, warranting its own discussion, than just bumping to 2.18 was. (I realize this PR is intended for that discussion!)

So far, I’m not sure there’s a single post‐2.18 language feature that we’ve had Nix and Lix buy‐in on stably exposing the same semantics for, beyond edge‐case semantic changes like integer overflow behaviour. (Maybe there is some built‐in supported in recent Nix and recent Lix that could be feature‐detected? I’ve mostly seen the Lix side taking issues with the design of new Nix built‐ins, but it’s entirely possible there is at least one thing there.)

To be clear, I’m not really opposed to the PR in the current form, I think it’s fine – I think we just differ on whether it makes a helpful difference in terms of our communication, and whether we’d even be doing the things it does at all (like checking for 2.18) in an era where we communicate an explicit list of supported implementation releases or required features.

Well, I agree with “not testing things not available in Nixpkgs”, at least; I don’t know if it makes sense to spend a ton of eval resources on checking a bunch of separate Lix versions for every PR in the merge queue, for instance – and anything we don’t do in the merge queue is a second tier of support already.

To be clear: I would not want the current "matrix of versions" that we test in updates to ci/pinned to happen in the merge queue. That's far too much in terms of resources. So "testing all versions" would always be "second tier" in that regard, because the checks for it are delayed until the next pinned update. We could, however, either run both Nix and Lix in the merge queue or run one of these in the merge queue and the other in PRs (where we still need Eval for labeling anyway). The former would still double the required resources, so I prefer the latter. Making both of these required and comparing the results (we already do) should give us the same kind of "test two branches, assume the fork point is likely good as well" verification for Nix 2.18. I think this would be a very good result already and a big improvement over what had been the status quo until recently.

We could, however, either run both Nix and Lix in the merge queue or run one of these in the merge queue and the other in PRs (where we still need Eval for labeling anyway). The former would still double the required resources, so I prefer the latter. Making both of these required and comparing the results (we already do) should give us the same kind of "test two branches, assume the fork point is likely good as well" verification for Nix 2.18. I think this would be a very good result already and a big improvement over what had been the status quo until recently.

I don’t think we can compare results between something done on a PR and something done on a later queue entry, as they will have different bases?

I don’t think we can compare results between something done on a PR and something done on a later queue entry, as they will have different bases?

The idea would be to:

• Run Lix in the Merge Queue.
• Run Nix in the PR.
• Don't run anything on push anymore.

The PR run compares against "the target branch". The commit the "test merge commit" merges into is the last commit on the target branch. The Eval run we use for comparison is the Eval run from the Merge Queue - but ofc, not from this PR, but from whatever was last merged into the target branch.

This results in the same kind of comparison we currently have, but with two versions compared against each other. And both of these are requirements before actually merging.

Edit: The "Nix and Lix differ in Eval" problem would not be a hard fail here, it would be a "soft" fail: This difference would start showing up in every PR as a rebuild. But that's not the thing we are primarily guarding against. The hard check is the fact that any change must Eval at all on both versions, not that it's exactly the same result.

The PR run compares against "the target branch". The commit the "test merge commit" merges into is the last commit on the target branch. The Eval run we use for comparison is the Eval run from the Merge Queue - but ofc, not from this PR, but from whatever was last merged into the target branch.

Doesn’t this mean that a PR “a: 1 → 2” merged into a branch that gained a commit “b: 3 → 4” since the CI run on the PR would be considered to differ between Nix and Lix, because the Nix eval has “b: 4 → 3” relative to the Lix one?

(Maybe we should move this to another issue though.)

Doesn’t this mean that a PR “a: 1 → 2” merged into a branch that gained a commit “b: 3 → 4” since the CI run on the PR would be considered to differ between Nix and Lix, because the Nix eval has “b: 4 → 3” relative to the Lix one?

No, the logic that we currently use stays the same: We always compare Eval results between the test merge commit and its first parent. It's just that the parent's Eval results have been created via push event before, but then will be via the merge_queue event.

Look at it as two separate changes:

• We first change Nix to Lix on push events. PR events stay the same. At this stage we run both Nix and Lix, and we also do some kind of comparison (the regular rebuilds). However, Lix is run after merge, so not a requirement.
• The second change moves the push event to the merge queue. This turns Lix into a requirement. The comparison stays the same.

At this point the minimum features list is effectively still just a minver

I mean.. we could just throw all builtins that we currently use in Nixpkgs in there for good measure. It won't make a practical difference, but:

• it would in theory allow linting via nixpkgs-vet based on that, so that we always keep the set of builtins in minfeatures up2date.
• it makes it clearer what the goal of this is.
• it would, in theory, allow entirely different implementations to be able to evaluate Nixpkgs - or be caught if they don't support all of that.

we could just throw all builtins that we currently use in Nixpkgs in there for good measure

I don't think we need to make this PR more controversial by doing that now, but it does sound like a good future step.

Being explicit about the required builtins doesn't help with required language features, but it is still a good direction to head in. I agree it makes the intent of minfeatures clearer.

I wonder if it would be possible to have CI that evaluates nixpkgs with an overridden/replaced/shadowed builtins, to enforce we only use builtins that are explicitly listed in minfeatures? Or maybe this could be done more easily via nixpkgs-vet?

I wonder if it would be possible to have CI that evaluates nixpkgs with an overridden/replaced/shadowed builtins, to enforce we only use builtins that are explicitly listed in minfeatures? Or maybe this could be done more easily via nixpkgs-vet?

Yeah, I think this is better done via static linting in nixpkgs-vet, which should require much fewer resources.

(I do agree on not adding it to this PR for sure)

Rebased over the conflict with #435683.

To be clear, I’m not really opposed to the PR in the current form, I think it’s fine – I think we just differ on whether it makes a helpful difference in terms of our communication, and whether we’d even be doing the things it does at all (like checking for 2.18) in an era where we communicate an explicit list of supported implementation releases or required features.

One thing that we didn't discuss, yet: This change might not communicate this to users, but it actually communicates it in-tree. Without this change, it would be easily possible to re-introduce some concepts like nixVersions.minimum, which is based on that "minimum version as a value". Equally, other checks across the tree could be introduced with this value present. With this change, this is not easily possible, everyone thinking along these lines has to think about the implications of minfeatures.nix already.

Since there were no objections so far, but two approvals, I'll merge it. If we find out we end up needing something else later on - then we can still change it. But I'd say this is the best idea we currently have for how it should be.