Skip to content

Conversation

hsjobeki
Copy link
Contributor

@hsjobeki hsjobeki commented Mar 18, 2025

For any kind of Introspection tooling we needed access to the definitions and the highestPrio of options.

Definitions via freeform are not accessible / observable at all which makes building debuggers, frontends, or introspection tools in general impossible. Also manual debugging by looking into freeform to see what was matched is nice and makes peoples life easier.

Overall:

  • This provides access for introspection of freeform, including 'prios' and 'definitions'

Thx @roberth who helped me out finding a design that works quite a bit.

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@github-actions github-actions bot added 6.topic: module system About "NixOS" module system internals 6.topic: lib The Nixpkgs function library labels Mar 18, 2025
@github-actions github-actions bot added 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Mar 18, 2025
@hsjobeki hsjobeki mentioned this pull request Mar 18, 2025
13 tasks
@hsjobeki hsjobeki marked this pull request as ready for review March 18, 2025 16:09
@nix-owners nix-owners bot requested review from roberth and infinisil March 18, 2025 16:11
@github-actions github-actions bot added the 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. label Mar 18, 2025
Copy link
Member

@infinisil infinisil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed together in a meeting with @hsjobeki. Overall looking fine!

  • _module.freeform should be renamed to something that clearly indicates what it is
  • It should also get some better docs
  • Why can't we add readOnly? (because mkMerge assigns multiple)
  • Why is the default necessary? (???)
  • Clean history

@github-actions github-actions bot removed the 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. label Mar 20, 2025
@hsjobeki hsjobeki changed the title lib.modules: Expose the freeform module via '_module.freeform' lib.modules: Expose the freeform module readonly via '_module.freeformConfig' Mar 20, 2025
@hsjobeki hsjobeki changed the title lib.modules: Expose the freeform module readonly via '_module.freeformConfig' lib.modules: Expose the freeform results readonly via '_module.freeformConfig' Mar 20, 2025
@wegank wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 2, 2025
Copy link
Member

@infinisil infinisil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not available a lot, so don't feel blocked on me, just left a very quick review :)

@ofborg ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 30, 2025
@hsjobeki hsjobeki requested a review from infinisil April 30, 2025 17:23
@github-actions github-actions bot removed the 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. label Apr 30, 2025
@github-actions github-actions bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. and removed 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Apr 30, 2025
@hsjobeki hsjobeki force-pushed the freeformDefs branch 2 times, most recently from 6ef0759 to adbdb8a Compare April 30, 2025 18:22
@github-actions github-actions bot added 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. and removed 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels Apr 30, 2025
@roberth
Copy link
Member

roberth commented May 9, 2025

LGTM, but let's not risk interrupting work on the release, so merge this after branch-off.
If that goes well, I'd support a backport when it's been on nixos-unstable for about a month.

@infinisil
Copy link
Member

infinisil commented Jul 3, 2025

After discussing this with @hsjobeki in a call today, we discovered that this PR currently doesn't work well for @hsjobeki's usecase, because while the freeform definitions would be exposed, the priorities of individual attributes would have to be recalculated outside the module evaluation. This would notably also be a problem for standard non-freeform options with nested priorities.

Instead, the valueMeta direction from #391544 allows this PR to work better because each merge (no matter how deep it occurs), can expose more detailed merge results, which could then be used for the freeform type merge as well, like this:

diff --git a/lib/modules.nix b/lib/modules.nix
index 4691018c68a0..611f6e46aac9 100644
--- a/lib/modules.nix
+++ b/lib/modules.nix
@@ -226,6 +226,10 @@ let
             '';
           };
 
+          _module.freeformMerged = mkOption {
+            internal = true;
+            visible = false;
+            readOnly = true;
+            type = types.raw;
+          };
+
           _module.specialArgs = mkOption {
             readOnly = true;
             internal = true;
@@ -242,6 +246,8 @@ let
             moduleType = type;
           };
           _module.specialArgs = specialArgs;
+
+          _module.freeformMerged = freeformMerged;
         };
       };
 
@@ -266,21 +272,26 @@ let
 
       options = merged.matchedOptions;
 
+      # For definitions that have an associated option
+      declaredConfig = mapAttrsRecursiveCond (v: !isOption v) (_: v: v.value) options;
+
+      freeformDefs = map (def: {
+        file = def.file;
+        value = setAttrByPath def.prefix def.value;
+      }) merged.unmatchedDefns;
+
+      freeformMerged = declaredConfig._module.freeformType.merge.v2 {
+        loc = prefix;
+        defs = freeformDefs;
+      };
+
       config =
         let
 
-          # For definitions that have an associated option
-          declaredConfig = mapAttrsRecursiveCond (v: !isOption v) (_: v: v.value) options;
 
           # If freeformType is set, this is for definitions that don't have an associated option
           freeformConfig =
-            let
-              defs = map (def: {
-                file = def.file;
-                value = setAttrByPath def.prefix def.value;
-              }) merged.unmatchedDefns;
-            in
-            if defs == [ ] then { } else declaredConfig._module.freeformType.merge prefix defs;
+            # Probably can be simplified with freeformMerged.isDefined or so?
+            if freeformDefs == [ ] then { } else freeformMerged.value;
 
         in
         if declaredConfig._module.freeformType == null then
diff --git a/lib/types.nix b/lib/types.nix
index 35171b28c799..07fa9d03e88b 100644
--- a/lib/types.nix
+++ b/lib/types.nix
@@ -852,6 +852,7 @@ let
                     v.optionalValue.value
                 ) evals;
                 valueMeta.attrs = mapAttrs (n: v: v.checkedAndMerged.valueMeta) evals;
+                valueMeta.evals = evals;
               };
           };
 

@roberth
Copy link
Member

roberth commented Jul 7, 2025

                 valueMeta.attrs = mapAttrs (n: v: v.checkedAndMerged.valueMeta) evals;
+                valueMeta.evals = evals;

This looks redundant or wrong.
What's the purpose of this?

+      freeformDefs = map (def: {

Floating this stuff out causes a space leak.
Although we have those all over the place, it's worth checking the impact on the final heap size.
Considering that this particular binding is only referenced by config (indirectly, but very much so), it'd be better to inline it, so that it can be released when freeformMerged has consumed it. (Best if .isDefined works, as commented)
Otherwise it will still be referenced through the scope that contains it, which is in turn likely referenced by any of the many thunks produced by evalModules.

@hsjobeki
Copy link
Contributor Author

hsjobeki commented Aug 13, 2025

+      freeformDefs = map (def: {

Floating this stuff out causes a space leak. Although we have those all over the place, it's worth checking the impact on the final heap size. Considering that this particular binding is only referenced by config (indirectly, but very much so), it'd be better to inline it, so that it can be released when freeformMerged has consumed it. (Best if .isDefined works, as commented) Otherwise it will still be referenced through the scope that contains it, which is in turn likely referenced by any of the many thunks produced by evalModules.

* [Naive thunk closures cause unnecessary space leaks nix#8285](https://github.com/NixOS/nix/issues/8285)

Thanks, thats good to know. i'll try move it back into the let binding again after we are done with #391544

We've basically discovered, that it would be better to integrate with merge.v2 because that would give us the new merge interface that natively allows exposing metadata

@nixpkgs-ci nixpkgs-ci bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Aug 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.status: merge conflict This PR has merge conflicts with the target branch 6.topic: lib The Nixpkgs function library 6.topic: module system About "NixOS" module system internals 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants