I'd like to provide some insights why we think using nix-unit is better than using the current lib/tests/modules.sh
And better than lib.runTest

• much faster (estimate some factor ~100).
• allows to load the test for debugging into the nix repl as it is just an attribute set.
• allows to organize tests in nested attribute sets.
• Allows to also assert on the error type not only the message.
• Expressions don't need to be matched as string.

Those and further benefits can also be found on the project page: https://github.com/nix-community/nix-unit

We currently run the suite against multiple Nix versions. If we have a pairing of nix-unit for each of them, I suppose we can continue to do so.

allows to load the test for debugging into the nix repl as it is just an attribute set.

Does that include intermediate values, like the whole evalModules result for example?
In the example, it seems that the only exposed values are the ones that are compared.

Another thing is that in the test suite we often use the same module for multiple assertions.
I suppose a let binding could help with that, and I would be inclined to model this pattern as:

# group for a specific but large module
intTypes =
  let
    inherit (evalModules ...) config;
  in
    {
      # test cases for each assertion
      negativeInt = ... config.negativeInt ...;
    };

Now I'm not sure that this is even the right pattern, as it's probably better to keep the test cases small and focused, so I guess that's something to pay attention to.
I don't think we'd want to "mechanically" translate the whole thing without cleaning it up.
But doing that is also risky, because the intent of the test may not always be clear, and therefore, mistakes can be made when shrinking the context of an assertion, making the assertion ineffective or less effective by not failing.
So I guess we do want a mechanical translation anyway, certainly at first.
Test cases can be refactored later when someone dives into the subject and thoroughly understands the intricacies.

a pairing of nix-unit for each of them

This could be done by having version conditionals using nix.version in the nix-unit package function.
Even src could be conditional. This way we don't have to add multi-version support upstream with CPP conditionals.

That way the existing lib/tests/release.nix stuff can create nix-units by calling nix-unit.override { inherit nix; }.

I just pushed a commit with all tests translated. While doing so i didn't change any of the tests although there where about 10 that had minor discrepancies in the regex behavior. We can see that translation is relatively trivial and that the result is absolutely identical. Despite we lack the ability for now to test against different nix versions.

/nixpkgs/lib/tests/modules/ > time nix-unit tests.nix > /dev/null
warning: unknown setting 'allowed-users'
warning: unknown setting 'trusted-users'
warning: `--gc-roots-dir' not specified
trace: { a = "one"; b = "two"; class = { just = "data"; }; meta = "meta"; }
nix-unit tests.nix > /dev/null  0,12s user 0,02s system 94% cpu 0,152 total
/nixpkgs/lib/tests/modules/ > time sh ../modules.sh
====== module tests ======
262 Pass
0 Fail
sh ../modules.sh  4,77s user 2,32s system 104% cpu 6,822 total

Pros:

• Debugging is easy (thus writing tests is easy)
• Organizing tests is easy.
• Speed: execution time is very fast
• Utility to provide coverage feedback exists
• Simple: Eval test are plain nix. No bash wrapper required.

Cons:

• Uses a static nix version currently
• Code becomes longer

Maybe we can find a suitable way of providing nix-unit with more than one nix version.
Q: Which versions do we need to support ?

Q: Which versions do we need to support ?

3 of them are currently being tested:

I think this is really neat overall, but testing with multiple Nix versions is fairly important. However I think it would be fine to drop nixVersions.minimum (2.3), because that version is mainly just needed to allow evaluating Nixpkgs to upgrade Nix itself.

@infinisil looking at the bigger picture, don't we want to use this infrastructure for all of lib's unit tests?
I also doubt that the minimum version shouldn't apply to NixOS, which uses the module system.
I guess it may be possible as a user to wiggle out of such a situation by first updating the nix client, with nix-shell -p, but this would be annoying and not at all obvious to users, even if they are few on 2.3.

Relatedly, we should certainly add the NixOS-stable default version as well. I believe that might currently coincide with nix, so we might be ok on that for now.