@Ma27, the commit nixos/system-path: allow users to override priority for requiredPackages exists so that one can easily override the priority of requiredPackages without having to do it manually for every package. I also wanted to find what paths are conflicting by setting the ignoreCollisions attribute's value to false but found no way to override it (like a derivation) and that is why commit nixos/system-path: make system.path overridable exists. I tried my best to explain this in the commit messages themselves.

Honestly this just feels wrong? Maybe we should just default priority to 0, and disallow collisions? At least that way the end users will hopefully be forced to choose.

the commit nixos/system-path: allow users to override priority for requiredPackages exists

Ouch, apologies!
I guess my heuristic of "nothing in the PR comment -> no details in the commit either" was at fault here (and yes, that's a me problem).

This means, if there were ever to be a conflict of paths between
rpi-imager and xz, rpi-imager would have a higher priority.

So, I think there are two cases where you run into this:

• if you choose a different implementation (e.g. GNU coreutils vs. uutils). Then doing hiPrio uutils-coreutils is IMHO the correct thing to override.

• cases like rpi-imager vs xz: IMHO these sound like a packaging bug. I'd be very curious to learn about the reason why this cannot use the xz from pkgs.xz but needs to ship its own. I think this is kind of the job of a packager to clean these things up (and in this case the primary target is NixOS where xz is available by default even).

  So for me, the motivation seems like a giant workaround for a bug.

This is just one instance of such an error that I only happened to
notice because of a guide I was writing.

This is probably the interesting thing to discuss about: were the collisions you encountered accidental (such as the xz thing) or was there a different reason? Right now, I'm not aware of a third case which is the reason for my skepticism.

Instead of directly assigning a derivation to system.path, a "dummy"
option now exists (system.pathDrv) which contains the initial
system-path derivation.

I'm wondering if adding a environment.ignoreCollisions (or in whatever else namespace) would be better just like we do for e.g. environment.extraOutputsToInstall. I don't see much of a reason to override random buildEnv-attributes and this would also mean that there are two ways now to do the exact same thing (i.e. specifying which outputs to install).

I agree that my commits don't actually solve the problem and feel wrong but I made them this way so that the user has more control over the knobs.

Another reason, as I mentioned in the first commit, is that if I come up with a PR to address this tree-wide, there's a very high probably it will not be a productive discussion. This is a temporary fix, until someone else can address it with more knowledge than me.

This means, if there were ever to be a conflict of paths between
rpi-imager and xz, rpi-imager would have a higher priority.

So, I think there are two cases where you run into this:

* if you choose a different implementation (e.g. GNU coreutils vs. uutils). Then doing `hiPrio uutils-coreutils` is IMHO the correct thing to override.

I agree. But IMHO, the packages in requiredPackages should have a higher priority and their output paths should not be overridden by any other packages.

This is just one instance of such an error that I only happened to
notice because of a guide I was writing.

This is probably the interesting thing to discuss about: were the collisions you encountered accidental (such as the xz thing) or was there a different reason? Right now, I'm not aware of a third case which is the reason for my skepticism.

I don't follow. I explained why I thought the collision occurred. I came to the conclusion that due to the current state, xz had the lower priority of 8 while rpi-imager had a higher priority of 5 and since the ${pkg}/bin/xz path collided, the higher priority of rpi-imager was used to link sw/bin/xz to ${pkgs.rpi-imager}/bin/xz instead of ${pkgs.xz}/bin/xz.

I don't know what you mean by "accidental." But I do believe that setting a lower priority for requiredPackages is plain wrong. They should either have the same priority as all other packages (to figure out collisions) or have a higher priority. The first commit allows one to set a negative value (say -15 for example) to ensure that all required packages have a high priority.

Instead of directly assigning a derivation to system.path, a "dummy"
option now exists (system.pathDrv) which contains the initial
system-path derivation.

I'm wondering if adding a environment.ignoreCollisions (or in whatever else namespace) would be better just like we do for e.g. environment.extraOutputsToInstall. I don't see much of a reason to override random buildEnv-attributes and this would also mean that there are two ways now to do the exact same thing (i.e. specifying which outputs to install).

Yeah that sounds better than my approach. I reworked the second commit and force pushed it.

Honestly this just feels wrong? Maybe we should just default priority to 0, and disallow collisions? At least that way the end users will hopefully be forced to choose.

Yes, but that would cause a lot of breakage. And I'm not familiar enough with the entirety of nixpkgs to check for every breakage and fix it treewide.

I don't follow. I explained why I thought the collision occurred. I came to the conclusion that due to the current state, xz had the lower priority of 8 while rpi-imager had a higher priority of 5 and since the ${pkg}/bin/xz path collided, the higher priority of rpi-imager was used to link sw/bin/xz to ${pkgs.rpi-imager}/bin/xz instead of ${pkgs.xz}/bin/xz.

My point is, the fact that rpi-imager has bin/xz in the first place sounds like a bug.
You were mentioning that this is just a single example amongst multiple issues. So I'm curious if all the other issues you faced could be classified as bug either.

If that's the case, I'd argue that this is the wrong knob, but instead we should establish explicit rules to not do this (unless they already exist and I just failed to find them) and fix the packages instead.

I agree. But IMHO, the packages in requiredPackages should have a higher priority and their output paths should not be overridden by any other packages.

Agreed, that's what you should use hiPrio for.
But then again, you still have requiredPriorityOffset = 3 by default which means that this isn't fixed, but people can fix it themselves easier, but with knowledge about package priorities required.

I'm not sure about what a sensible migration path would be, but I think I agree with @K900 here on how to fix this, i.e. by changing the requiredPackages-priority.

I get your point. Would it be better if the first commit were to set a priority value of 0 for all (requiredPackages and config.environment.systemPackages) packages? And the second commit were to set options.environment.ignoreCollisions.default to false?

I'll also work on finding and resolving as many conflicts as I can.

Sounds reasonable to me. @K900 wdyt?

also cc @anthonyroussel for rpi-imager.

Sounds reasonable.