lib.modules: init types checkAndMerge to allow adding 'valueMeta' #391544

hsjobeki · 2025-03-20T11:42:35Z

This changes the merge behavior of some types to return an enveloping attribute set instead of the plain value

{
 value = ...;
 # Information about the value. Such as errors, submodule evaluations, attrs, list items, etc. (depends on what is important in the context of this type)
 valueMeta = { ... } ;
}

Some examples:

myEval.options.fileSystems.valueMeta.attrs."/".options
# i.e. sub-options of 'attrsOf submodule' are now actually traversable

This is related to #390952 which exposes the freeform module. After both PRs are merged the module-system should be mostly introspectable within the language and thus also by LSPs.

cc: @inclyc Do you want to integrate this into nixd after its merged?

Things done

Add merge enveloping to attrsOf value = ...; valueMeta = { attrs }
Add merge enveloping to listOf value = ...; valueMeta = { list }
Add merge enveloping to submoduleWith value = ...; valueMeta = { config, options, .... }
migrate merge of types.either
migrate merge of types.coercedTo
Utility for backwards compatible defs checking in terms of the new system checkDefsForError

TODO for proof of stability

Regression test against different lib versions (see Cross compatibility)
- evalModules from this version against types from an older version
- evalModules from an older version against types from this version
Manual Performance Comparison of two plain nixos (CI doesn't offer that yet unfortunately) (see Performance comparison)
Performance Comparison if we maintain duplicate v1 code (maybe omit that, only relevant for people using prior evalModules and/or prior types against newer evalModule, newer types; see reply )

Performance comparison (plain nixos)

Expand

nixos = import ./nixos/lib/eval-config.nix {
  modules = [
    {
      fileSystems."/".device = "/dev/null";
      boot.loader.systemd-boot.enable = true;
    }
    {
      nixpkgs.system = "x86_64-linux";
    }
   ];
};

NIX_SHOW_STATS=1 NIX_SHOW_STATS_PATH=stats.json nix-instantiate ./t.nix -A 'nixos.config.system.build.toplevel' --eval-only > /dev/null

I did a run with hyperfine 20 runs with 1 warmup.
CPU stats and time is noisy.

Key	Before (μ ± SEM)	After (μ ± SEM)	Δ Abs	Δ %
cpuTime	3.474885179882958 ± 0.024193601265817277	3.529527868543352 ± 0.0211241453405698	0.054643	1.57%
envs.bytes	99463520 ± 0.0	100558632 ± 0.0	1095112.000000	1.10%
envs.elements	`7149072` ± 0.0	7258437 ± 0.0	109365.000000	1.53%
envs.number	5283868 ± 0.0	5311392 ± 0.0	27524.000000	0.52%
gc.cycles	3 ± 0.0	3 ± 0.0	0.000000	0.00%
gc.heapSize	520810496 ± 0.0	520794112 ± 12494.318721113284	-16384.000000	-0.00%
gc.totalBytes	757498767.2380953 ± 1114.0300626548938	764052354.2857143 ± 1005.967587149088	6553587.047619	0.87%
list.bytes	12730752 ± 0.0	`1272286` ± 0.0	-7888.000000	-0.06%
list.concats	159362 ± 0.0	159362 ± 0.0	0.000000	0.00%
list.elements	1591344 ± 0.0	1590358 ± 0.0	-986.000000	-0.06%
nrAvoided	5341771 ± 0.0	5359965 ± 0.0	18194.000000	0.34%
nrExprs	1408676 ± 0.0	1408977 ± 0.0	301.000000	0.02%
nrFunctionCalls	4602247 ± 0.0	4617834 ± 0.0	15587.000000	0.34%
nrLookups	2839645 ± 0.0	2881290 ± 0.0	41645.000000	1.47%
nrOpUpdateValuesCopied	10480360 ± 0.0	10499851 ± 0.0	19491.000000	0.19%
nrOpUpdates	306034 ± 0.0	306096 ± 0.0	62.000000	0.02%
nrPrimOpCalls	2152319 ± 0.0	2150493 ± 0.0	-1826.000000	-0.08%
nrThunks	6327944 ± 0.0	6434178 ± 0.0	106234.000000	1.68%
sets.bytes	268536048 ± 0.0	270352560 ± 0.0	1816512.000000	0.68%
sets.elements	15456314 ± 0.0	15554996 ± 0.0	98682.000000	0.64%
sets.number	1327189 ± 0.0	1342039 ± 0.0	14850.000000	1.12%
sizes.Attr	16 ± 0.0	16 ± 0.0	0.000000	0.00%
sizes.Bindings	16 ± 0.0	16 ± 0.0	0.000000	0.00%
sizes.Env	8 ± 0.0	8 ± 0.0	0.000000	0.00%
sizes.Value	24 ± 0.0	24 ± 0.0	0.000000	0.00%
symbols.bytes	1038809 ± 0.0	1038915 ± 0.0	106.000000	0.01%
symbols.number	85670 ± 0.0	85678 ± 0.0	8.000000	0.01%
time.cpu	3.474885179882958 ± 0.024193601265817277	3.529527868543352 ± 0.0211241453405698	0.054643	1.57%
time.gc	1.235095238095238 ± 0.008809601029262603	1.297047619047619 ± 0.016255764719671457	0.061952	5.02%
time.gcFraction	0.3556172479196542 ± 0.002634608218747617	0.36726246541562935 ± 0.0030404671592494255	0.011645	3.27%
values.bytes	251794584 ± 0.0	254430240 ± 0.0	2635656.000000	1.05%
values.number	10491441 ± 0.0	10601260 ± 0.0	109819.000000	1.05%

If you got any ideas how to further improve the performance, i'm more than thankfull

Cross compatibility

Expand

Test setup:
I modified the lib/tests/modules/default.nix

We also need to disable all the valueMeta tests, they are not expected to work if either the type or the evalModules doesn't support it.

{
  lib ? import ../..,
  modules ? [ ],
}:
let
  thisLib = lib;
  pkgs = import <nixpkgs> {};
  prevNixpkgs = pkgs.fetchgit {
    url = "https://github.com/nixos/nixpkgs.git";
    rev = "98ce1f5ab866f073a94bfa7b0408f83c926a6202";
    hash = "sha256-XbisKB5blkuFbL5EQMVNwRmhQzIZLcXfPtw2zP4dFcM=";
  };
  prevLib = (import prevNixpkgs { }).lib;
in
{
  inherit
    (prevLib.evalModules {
      inherit modules;
      specialArgs.modulesPath = ./.;
      # specialArgs has the highest prio
      # I use that to override the version of 'lib' that the test modules use
      # Example: { lib , ... }: { ... some test }
      specialArgs.lib = thisLib;
    })
    config
    options
    ;
}

We can switch around prevLib.evalModules and sepcialArgs.lib = thisLib
This should work for all test modules that get their lib.types from the modules arguments like { lib, ... }: { ... test ... }

This allows me to run our test suite with mismatching versions.

Forward compatibility results:

(old) evalModules;
(new) types;

$> ./modules.sh 
====== module tests ======
343 Pass
0 Fail

Backward compatibility results:

(new) evalModules;
(old) types;

$> ./modules.sh 
====== module tests ======
343 Pass
0 Fail

Open questions:

How to implement addCheck ? It somehow needs to inject the checking behavior into merge...

Add a 👍 reaction to pull requests you find important.

inclyc · 2025-03-21T04:04:33Z

cc: @inclyc Do you want to integrate this into nixd after its merged?

Apologies, but I haven’t fully grasped the specific improvements introduced by this PR or what LSPs would be able to achieve once it’s merged. Could you provide some insights with a step-by-step example in nix-repl?

hsjobeki · 2025-03-23T10:40:45Z

cc: @inclyc Do you want to integrate this into nixd after its merged?

Apologies, but I haven’t fully grasped the specific improvements introduced by this PR or what LSPs would be able to achieve once it’s merged. Could you provide some insights with a step-by-step example in nix-repl?

Of course sorry for the confusion.
Here is an example:

# Lets create a simple nixosSystem evaluation like this.
myEval = nixos { fileSystems."/" = {}; } 
#   'options.valueMeta': now contains meta Information about the value
#   i.e. types.attrsOf has 'attrs'. 
#   Below is an attrsOf submodule option which allows access to the sub options (also recursively) now.
nix-repl> myEval.options.fileSystems.valueMeta.attrs."/".options
{
  _module = { ... };
  autoFormat = { ... };
  autoResize = { ... };
  depends = { ... };
  device = { ... };
  enable = { ... };
  encrypted = { ... };
  formatOptions = { ... };
  fsType = { ... };
  label = { ... };
  mountPoint = { ... };
  neededForBoot = { ... };
  noCheck = { ... };
  options = { ... };
  overlay = { ... };
  stratis = { ... };
}

Previously this option had only myEval.options.fileSystems.getSubOptions loc. Which was a function and would only return staticModules which are incomplete and wouldn't reveal the actual information.

Now you can introspect i.e. highestPrio or value (or whatever you like) within the option tree without any hassle.

hsjobeki · 2025-04-30T19:35:47Z

@roberth @infinisil I've rebase this PR and added some tests

wolfgangwalther · 2025-05-11T10:24:22Z

Needs another rebase to get rid of ci/compare: nix stats comparison.

roberth

This is a very significant PR, and it needs some issues to be addressed before we can unleash this onto the ecosystem.

Small clarification on `checkAndMerge`

Something like checkAndMerge is required for the (efficient) propagation of "metadata" through the type interface.
Concretely, this is because the computation work of e.g. submodule happens through the merge function, which can only return the removeAttrs config ["_module"] value, as part of the submodule contract. (otherwise we'd be spelling out things like config.fileSystems."/".config.device all the time, which would be unpleasant, let's say)

In order to achieve the intended functionality - accessing things like options anywhere in the configuration - we have no choice but to create an alternative to merge.

Changing the type interface

The type interface (_type = "optionType";) is an implicit interface that's made somewhat explicit by mkOptionType, but this function has very limited control over the types it produces in practice, because it is fairly common practice to refine existing types into new types using the // operator.

This creates a lack of control that makes it impossible to handle such cases in an architecturally pleasant way, and somewhat difficult to handle altogether.

In order not to break such usages, the burden of handling both the new and old methods (merge vs checkAndMerge) falls on the caller, as we have relatively few direct callers of the type interface, and even fewer where the distinction between these methods is all that relevant.
We should make sure that everything users want to do with // can be done with helper functions instead, and when that's done, we can start warning about the // usages we can detect.

When @hsjobeki and I worked on this during OceanSprint, we originally tried to have a nice and consistent type interface, which always has a checkAndMerge function, but this would obstruct our ability to detect usages such as // { check = ... }; and handle them accordingly.
(See also the review comment on mkOptionType)

`check` functionality

This part of the solution is still underdeveloped, and needs some more exploration before I'd feel comfortable approving these changes. (It's a bit of a research-y PR)

check serves multiple purposes.

It is applied to each definition to make sure mistakes are reported nicely in the context of the module where the mistake was made, as opposed to just an option path
It is used by types.either and types.coercedTo to decide which type to delegate to.

It being a separate function is somewhat problematic

People make wrong assumptions about it
It can not share computation with the merge functionality

Furthermore, since the check interface (definition value -> bool) is very limited, the error reporting is actually not great. No custom error message are possible.

This PR currently does not do much to alleviate these problems, and it may muddy the water for future such improvements. I believe this should be part of the PR's scope, unfortunately, because delaying it vastly increases the cost of doing it, as well as posing a risk of committing to a mistake that causes yet more complexity to handle all the "versions" of the type interface correctly (noting that users write their own types too, and multiple versions of lib occasionally interact).

I believe the replacement of check should be an attribute in the checkAndMerge result. This allows the definition checking logic and merging logic to share a closure, which may have a positive effect on performance for the same reason as why value and valueMeta or combined into that attrset.
It might also positively affect taggedSubmodule. (unsure)
Due to checkAndMerge receiving a list of definitions, this check, this attribute needs to check a list of definitions. This should usually be done with a helper function.
Instead of returning false, this attribute could return an error string, and null for the current check x == true case. (better than true because a string isn't a valid if condition anyway, so let's not make a wrong impression. Also nullOr str is a good model for it, nicer than either str (enum [false]))

Naming

We settled on checkAndMerge for the time being, but it's somewhat misleading.
It does capture a high-level essence of the purpose of the function, but I believe we'd be better off with a more low-level name that only describes what it does, and not what its attributes do. This then helps create the correct mental model for what it is and how it behaves.
I'm thinking something like:

# old checkAndMerge
withDefinitions = loc: defs: {
  # old check, but an error string instead.
  # add "head" to signal that it's not a deep check, or at least usually isn't
  # this also leaves space for a possible deep `error` attribute, a bit like `tryEval (deepSeq value)` but without `tryEval` and able to return an error message as a string. (out of scope for this PR)
  headError = <...>;

  value = <...>;
  
  valueMeta = <...>;
}

roberth · 2025-05-12T08:19:20Z

lib/types.nix

@@ -204,6 +205,10 @@ let
        # definition values and locations (e.g. [ { file = "/foo.nix";
        # value = 1; } { file = "/bar.nix"; value = 2 } ]).
        merge ? mergeDefaultOption,


In order to properly detect "misuse" of // { check } onto a checkAndMerge-based type, we may have to make check and merge optional too.

Removing check from type constructors like attrsOf and submodule is probably a good thing, because when users rely on the check function of these types, they tend to be unaware that it only checks the head, i.e. isAttrs, as the rest is (necessarily) checked in merge, not check.

addCheck should then also propagate the missing-ness of check.

This should make headError extensible other information needs to be passed This seems to improve performance slightly

… set

Mic92 · 2025-08-13T13:15:29Z

Some real-world testing: Mic92/dotfiles#3279

hsjobeki · 2025-08-14T12:36:04Z

Just updated the performance report in the description again. I think the performance is within the acceptable margin.

infinisil

Looked at this together with @hsjobeki again today and I'm pretty happy with this PR as is, #391544 (comment) is the only pending topic really. @roberth could you give your approval so we can merge it?

roberth · 2025-08-14T15:56:35Z

I'm still worried about check or headError not being called as is needed.

It wasn't part of our original draft implementation
The test suite needs to cover multiple combinations in a Cartesian product of
- outer call is
  - v1
  - v2
- outer type is
  - attrsOf
  - listOf
  - submodule
  - ...
- inner type supports
  - v1
  - both

I can't easily tell whether that's all covered

hsjobeki · 2025-08-15T09:58:07Z

I'm still worried about check or headError not being called as is needed.

* It wasn't part of our original draft implementation

* The test suite needs to cover multiple combinations in a Cartesian product of
  
  * outer call is
    
    * v1
    * v2
  * outer type is
    
    * attrsOf
    * listOf
    * submodule
    * ...
  * inner type supports
    
    * v1
    * both

I can't easily tell whether that's all covered

Hm... we'd need some custom test framework to test these combinations 🤔
To eliminate possible breakages i manually run:

(old) evalModules;
(new) types;

And the other way around as described in the PR description. However i couldn't get this to work with our CI.
Maybe this would be required to ensure backwards compat to the branch of point? Though it would require a completely new test setup that allows testing lib for cross compatibility in general.
Say: use a nested type from commit X in a submodule type from commit Y. and the other way around, this might be more common in the wild than we think.

We dont have the testing infrastructure yet to test this. If you prefer i could do either:

A: Test setup to run manually ( i.e. preferably a test suite in nix-unit that you can also execute)
B: CI tests to ensure we always stay compatible with the parent of the PR commit. Would be more effort to integrate with nixpkgs ci; not sure if worth it. but maybe required for long term stability ?

@infinisil @roberth tell me what you prefer or what you think is necessary.

Thinking for myself i think going for B is the correct thing to do...
Though it requires inventing a new way of testing lib

hsjobeki · 2025-08-15T11:57:20Z

Okay.. back from writing unit tests in nix-unit. I fetched an older lib version and run 144 Tests using the requested matrix.

Here is the list of my nix-unit tests (will try to hook up with our ci now)

2 * 2 * 2 = 8 tests ; multiplied by about ~= 1 ok case and 2-4 error cases to test headError and checking infrastructure depending on the type

Expand

✅ attrsOf_str_err_inner.call_v1.outer_v1.inner_v1.test_attrsOf_str
✅ attrsOf_str_err_inner.call_v1.outer_v1.inner_v2.test_attrsOf_str
✅ attrsOf_str_err_inner.call_v1.outer_v2.inner_v1.test_attrsOf_str
✅ attrsOf_str_err_inner.call_v1.outer_v2.inner_v2.test_attrsOf_str
✅ attrsOf_str_err_inner.call_v2.outer_v1.inner_v1.test_attrsOf_str
✅ attrsOf_str_err_inner.call_v2.outer_v1.inner_v2.test_attrsOf_str
✅ attrsOf_str_err_inner.call_v2.outer_v2.inner_v1.test_attrsOf_str
✅ attrsOf_str_err_inner.call_v2.outer_v2.inner_v2.test_attrsOf_str
✅ attrsOf_str_err_outer.call_v1.outer_v1.inner_v1.test_attrsOf_str
✅ attrsOf_str_err_outer.call_v1.outer_v1.inner_v2.test_attrsOf_str
✅ attrsOf_str_err_outer.call_v1.outer_v2.inner_v1.test_attrsOf_str
✅ attrsOf_str_err_outer.call_v1.outer_v2.inner_v2.test_attrsOf_str
✅ attrsOf_str_err_outer.call_v2.outer_v1.inner_v1.test_attrsOf_str
✅ attrsOf_str_err_outer.call_v2.outer_v1.inner_v2.test_attrsOf_str
✅ attrsOf_str_err_outer.call_v2.outer_v2.inner_v1.test_attrsOf_str
✅ attrsOf_str_err_outer.call_v2.outer_v2.inner_v2.test_attrsOf_str
✅ attrsOf_str_ok.call_v1.outer_v1.inner_v1.test_attrsOf_str
✅ attrsOf_str_ok.call_v1.outer_v1.inner_v2.test_attrsOf_str
✅ attrsOf_str_ok.call_v1.outer_v2.inner_v1.test_attrsOf_str
✅ attrsOf_str_ok.call_v1.outer_v2.inner_v2.test_attrsOf_str
✅ attrsOf_str_ok.call_v2.outer_v1.inner_v1.test_attrsOf_str
✅ attrsOf_str_ok.call_v2.outer_v1.inner_v2.test_attrsOf_str
✅ attrsOf_str_ok.call_v2.outer_v2.inner_v1.test_attrsOf_str
✅ attrsOf_str_ok.call_v2.outer_v2.inner_v2.test_attrsOf_str
✅ attrsOf_submodule_err_inner.call_v1.outer_v1.inner_v1.test_attrsOf_submodule
✅ attrsOf_submodule_err_inner.call_v1.outer_v1.inner_v2.test_attrsOf_submodule
✅ attrsOf_submodule_err_inner.call_v1.outer_v2.inner_v1.test_attrsOf_submodule
✅ attrsOf_submodule_err_inner.call_v1.outer_v2.inner_v2.test_attrsOf_submodule
✅ attrsOf_submodule_err_inner.call_v2.outer_v1.inner_v1.test_attrsOf_submodule
✅ attrsOf_submodule_err_inner.call_v2.outer_v1.inner_v2.test_attrsOf_submodule
✅ attrsOf_submodule_err_inner.call_v2.outer_v2.inner_v1.test_attrsOf_submodule
✅ attrsOf_submodule_err_inner.call_v2.outer_v2.inner_v2.test_attrsOf_submodule
✅ attrsOf_submodule_err_outer.call_v1.outer_v1.inner_v1.test_attrsOf_submodule
✅ attrsOf_submodule_err_outer.call_v1.outer_v1.inner_v2.test_attrsOf_submodule
✅ attrsOf_submodule_err_outer.call_v1.outer_v2.inner_v1.test_attrsOf_submodule
✅ attrsOf_submodule_err_outer.call_v1.outer_v2.inner_v2.test_attrsOf_submodule
✅ attrsOf_submodule_err_outer.call_v2.outer_v1.inner_v1.test_attrsOf_submodule
✅ attrsOf_submodule_err_outer.call_v2.outer_v1.inner_v2.test_attrsOf_submodule
✅ attrsOf_submodule_err_outer.call_v2.outer_v2.inner_v1.test_attrsOf_submodule
✅ attrsOf_submodule_err_outer.call_v2.outer_v2.inner_v2.test_attrsOf_submodule
✅ attrsOf_submodule_ok.call_v1.outer_v1.inner_v1.test_attrsOf_submodule
✅ attrsOf_submodule_ok.call_v1.outer_v1.inner_v2.test_attrsOf_submodule
✅ attrsOf_submodule_ok.call_v1.outer_v2.inner_v1.test_attrsOf_submodule
✅ attrsOf_submodule_ok.call_v1.outer_v2.inner_v2.test_attrsOf_submodule
✅ attrsOf_submodule_ok.call_v2.outer_v1.inner_v1.test_attrsOf_submodule
✅ attrsOf_submodule_ok.call_v2.outer_v1.inner_v2.test_attrsOf_submodule
✅ attrsOf_submodule_ok.call_v2.outer_v2.inner_v1.test_attrsOf_submodule
✅ attrsOf_submodule_ok.call_v2.outer_v2.inner_v2.test_attrsOf_submodule
✅ coerce_attrsOf_str_to_listOf_err_coercer_input.call_v1.outer_v1.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_input.call_v1.outer_v1.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_input.call_v1.outer_v2.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_input.call_v1.outer_v2.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_input.call_v2.outer_v1.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_input.call_v2.outer_v1.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_input.call_v2.outer_v2.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_input.call_v2.outer_v2.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_ouput.call_v1.outer_v1.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_ouput.call_v1.outer_v1.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_ouput.call_v1.outer_v2.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_ouput.call_v1.outer_v2.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_ouput.call_v2.outer_v1.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_ouput.call_v2.outer_v1.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_ouput.call_v2.outer_v2.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_err_coercer_ouput.call_v2.outer_v2.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_final.call_v1.outer_v1.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_final.call_v1.outer_v1.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_final.call_v1.outer_v2.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_final.call_v1.outer_v2.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_final.call_v2.outer_v1.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_final.call_v2.outer_v1.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_final.call_v2.outer_v2.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_final.call_v2.outer_v2.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_run.call_v1.outer_v1.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_run.call_v1.outer_v1.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_run.call_v1.outer_v2.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_run.call_v1.outer_v2.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_run.call_v2.outer_v1.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_run.call_v2.outer_v1.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_run.call_v2.outer_v2.inner_v1.test_coercedTo_attrsOf_str->listOf_str
✅ coerce_attrsOf_str_to_listOf_str_run.call_v2.outer_v2.inner_v2.test_coercedTo_attrsOf_str->listOf_str
✅ either_str_attrsOf_err_1.call_v1.outer_v1.inner_v1.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_1.call_v1.outer_v1.inner_v2.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_1.call_v1.outer_v2.inner_v1.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_1.call_v1.outer_v2.inner_v2.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_1.call_v2.outer_v1.inner_v1.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_1.call_v2.outer_v1.inner_v2.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_1.call_v2.outer_v2.inner_v1.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_1.call_v2.outer_v2.inner_v2.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_2.call_v1.outer_v1.inner_v1.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_2.call_v1.outer_v1.inner_v2.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_2.call_v1.outer_v2.inner_v1.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_2.call_v1.outer_v2.inner_v2.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_2.call_v2.outer_v1.inner_v1.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_2.call_v2.outer_v1.inner_v2.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_2.call_v2.outer_v2.inner_v1.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_err_2.call_v2.outer_v2.inner_v2.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_ok.call_v1.outer_v1.inner_v1.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_ok.call_v1.outer_v1.inner_v2.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_ok.call_v1.outer_v2.inner_v1.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_ok.call_v1.outer_v2.inner_v2.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_ok.call_v2.outer_v1.inner_v1.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_ok.call_v2.outer_v1.inner_v2.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_ok.call_v2.outer_v2.inner_v1.test_either_str_or_attrsOf_str
✅ either_str_attrsOf_ok.call_v2.outer_v2.inner_v2.test_either_str_or_attrsOf_str
✅ listOf_str_err_inner.call_v1.outer_v1.inner_v1.test_listOf_str
✅ listOf_str_err_inner.call_v1.outer_v1.inner_v2.test_listOf_str
✅ listOf_str_err_inner.call_v1.outer_v2.inner_v1.test_listOf_str
✅ listOf_str_err_inner.call_v1.outer_v2.inner_v2.test_listOf_str
✅ listOf_str_err_inner.call_v2.outer_v1.inner_v1.test_listOf_str
✅ listOf_str_err_inner.call_v2.outer_v1.inner_v2.test_listOf_str
✅ listOf_str_err_inner.call_v2.outer_v2.inner_v1.test_listOf_str
✅ listOf_str_err_inner.call_v2.outer_v2.inner_v2.test_listOf_str
✅ listOf_str_err_outer.call_v1.outer_v1.inner_v1.test_listOf_str
✅ listOf_str_err_outer.call_v1.outer_v1.inner_v2.test_listOf_str
✅ listOf_str_err_outer.call_v1.outer_v2.inner_v1.test_listOf_str
✅ listOf_str_err_outer.call_v1.outer_v2.inner_v2.test_listOf_str
✅ listOf_str_err_outer.call_v2.outer_v1.inner_v1.test_listOf_str
✅ listOf_str_err_outer.call_v2.outer_v1.inner_v2.test_listOf_str
✅ listOf_str_err_outer.call_v2.outer_v2.inner_v1.test_listOf_str
✅ listOf_str_err_outer.call_v2.outer_v2.inner_v2.test_listOf_str
✅ listOf_str_ok.call_v1.outer_v1.inner_v1.test_listOf_str
✅ listOf_str_ok.call_v1.outer_v1.inner_v2.test_listOf_str
✅ listOf_str_ok.call_v1.outer_v2.inner_v1.test_listOf_str
✅ listOf_str_ok.call_v1.outer_v2.inner_v2.test_listOf_str
✅ listOf_str_ok.call_v2.outer_v1.inner_v1.test_listOf_str
✅ listOf_str_ok.call_v2.outer_v1.inner_v2.test_listOf_str
✅ listOf_str_ok.call_v2.outer_v2.inner_v1.test_listOf_str
✅ listOf_str_ok.call_v2.outer_v2.inner_v2.test_listOf_str
✅ submodule_with_ok.call_v1.outer_v1.inner_v1.test_submoduleWith_mixed_types
✅ submodule_with_ok.call_v1.outer_v1.inner_v2.test_submoduleWith_mixed_types
✅ submodule_with_ok.call_v1.outer_v2.inner_v1.test_submoduleWith_mixed_types
✅ submodule_with_ok.call_v1.outer_v2.inner_v2.test_submoduleWith_mixed_types
✅ submodule_with_ok.call_v2.outer_v1.inner_v1.test_submoduleWith_mixed_types
✅ submodule_with_ok.call_v2.outer_v1.inner_v2.test_submoduleWith_mixed_types
✅ submodule_with_ok.call_v2.outer_v2.inner_v1.test_submoduleWith_mixed_types
✅ submodule_with_ok.call_v2.outer_v2.inner_v2.test_submoduleWith_mixed_types

To ensure forward and backward stability i am looking for a way to integrate this into our CI now

Needed to ensure backwards stability of types.merge.v2 added in NixOS#391544

hsjobeki · 2025-08-20T20:03:24Z

Replied to the last open question here:
#391544 (comment)

roberth

❤️

roberth · 2025-08-28T12:35:17Z

This was a big one.
Thank you @hsjobeki!

hsjobeki · 2025-08-30T09:24:29Z

Fyi: There is a minor regression on the either type. Fix is in #438558

hsjobeki requested a review from infinisil March 20, 2025 11:42

github-actions bot added 6.topic: module system About "NixOS" module system internals 6.topic: lib The Nixpkgs function library labels Mar 20, 2025

nix-owners bot requested a review from roberth March 20, 2025 11:44

github-actions bot added 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Mar 20, 2025

hsjobeki changed the title ~~lib.modules: init types checkAndMerge to allow adding 'outOfBand' att…~~ lib.modules: init types checkAndMerge to allow adding 'valueMeta' Mar 20, 2025

roberth mentioned this pull request Mar 30, 2025

Replace 'configurationsXyz' and 'modulesXyz' with 'configurations.xyz' and 'modules.xyz' NixOS/nix#6257

Open

hsjobeki force-pushed the modules-out-of-band branch from c3e038c to 860e45d Compare April 1, 2025 07:35

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 2, 2025

roberth mentioned this pull request Apr 7, 2025

lib/options: introduce mkSubmoduleAttrsOption #396742

Closed

13 tasks

hsjobeki force-pushed the modules-out-of-band branch from 860e45d to 6ee4232 Compare April 30, 2025 19:20

ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 30, 2025

hsjobeki force-pushed the modules-out-of-band branch 2 times, most recently from 78dc943 to 62bc75a Compare May 6, 2025 20:06

github-actions bot added 6.topic: continuous integration Affects continuous integration (CI) in Nixpkgs, including Ofborg and GitHub Actions backport release-24.11 labels May 6, 2025

nix-owners bot requested review from philiptaron, risicle, LeSuisse, mweinelt and wolfgangwalther May 6, 2025 20:08

hsjobeki force-pushed the modules-out-of-band branch from 62bc75a to 4685285 Compare May 6, 2025 20:27

roberth requested changes May 12, 2025

View reviewed changes

hsjobeki force-pushed the modules-out-of-band branch from 4685285 to 5f4875d Compare May 15, 2025 12:57

hsjobeki added 5 commits August 13, 2025 15:09

lib/modules: optimize performance by inlining bindings

ebafc3e

lib/modules: test revert unentional regression in check

1765370

lib/modules: add nested 'headError.message'

50bef19

This should make headError extensible other information needs to be passed This seems to improve performance slightly

types/merge: move 'configuration' of submodules into nested attribute…

cd2e5bd

… set

types/addCheck: add tests for merge v1 and v2

45ed757

hsjobeki force-pushed the modules-out-of-band branch from ab1e9b2 to 45ed757 Compare August 13, 2025 13:09

hsjobeki force-pushed the modules-out-of-band branch from 9144187 to 45ed757 Compare August 13, 2025 13:48

This comment was marked as resolved.

Sign in to view

infinisil approved these changes Aug 14, 2025

View reviewed changes

hsjobeki added a commit to hsjobeki/nixpkgs that referenced this pull request Aug 15, 2025

lib/tests: introduce lib cross version checks

8aebefc

Needed to ensure backwards stability of types.merge.v2 added in NixOS#391544

lib/tests: introduce lib cross version checks

1fed602

Needed to ensure backwards stability of types.merge.v2 added in NixOS#391544

hsjobeki force-pushed the modules-out-of-band branch from 8aebefc to 1fed602 Compare August 15, 2025 13:26

hsjobeki added 2 commits August 18, 2025 08:01

lib/modules: fix typo

4f802d9

lib/modules: add _internal to valueMeta of checkedAndMerged

bb0bd3d

hsjobeki requested review from roberth and infinisil August 20, 2025 19:55

ConnorBaker mentioned this pull request Aug 27, 2025

Structured warnings and assertions ConnorBaker/ideas#5

Open

roberth approved these changes Aug 28, 2025

View reviewed changes

roberth merged commit 83fed2e into NixOS:master Aug 28, 2025
26 of 27 checks passed

This was referenced Aug 30, 2025

nixos/cups-pdf: funny module system eval error #438459

Closed

lib/types: fix either lazy value regression in case of non-matching v… #438558

Open

Uh oh!

lib.modules: init types checkAndMerge to allow adding 'valueMeta' #391544

lib.modules: init types checkAndMerge to allow adding 'valueMeta' #391544

Uh oh!

Conversation

hsjobeki commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things done

Performance comparison (plain nixos)

Cross compatibility

Open questions:

Uh oh!

inclyc commented Mar 21, 2025

Uh oh!

hsjobeki commented Mar 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hsjobeki commented Apr 30, 2025

Uh oh!

wolfgangwalther commented May 11, 2025

Uh oh!

roberth left a comment

Choose a reason for hiding this comment

Small clarification on checkAndMerge

Changing the type interface

check functionality

Naming

Uh oh!

roberth May 12, 2025

Choose a reason for hiding this comment

Uh oh!

Mic92 commented Aug 13, 2025

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

hsjobeki commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

infinisil left a comment

Choose a reason for hiding this comment

Uh oh!

roberth commented Aug 14, 2025

Uh oh!

hsjobeki commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hsjobeki commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hsjobeki commented Aug 20, 2025

Uh oh!

roberth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

roberth commented Aug 28, 2025

Uh oh!

hsjobeki commented Aug 30, 2025

Uh oh!

Uh oh!

hsjobeki commented Mar 20, 2025 •

edited

Loading

hsjobeki commented Mar 23, 2025 •

edited

Loading

Small clarification on `checkAndMerge`

`check` functionality

hsjobeki commented Aug 14, 2025 •

edited

Loading

hsjobeki commented Aug 15, 2025 •

edited

Loading

hsjobeki commented Aug 15, 2025 •

edited

Loading