greetings @junjihashimoto this should target staging branch due to mass rebuilds

https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md#changes-causing-mass-rebuilds

beware of commit pitfall when changing target branch https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md#rebasing-between-branches-ie-from-master-to-staging

@kirillrdy Thx! Rebased to staging branch.

LGTM. I am able to use this branch to build a rust binary which I couldn't build with nixpkgs-unstable.

Interestingly this doesn't solve my issue trying to built qdrant 1.8.4, which contains two "wal 0.1.2"s with different git revs https://github.com/qdrant/qdrant/blob/v1.8.4/Cargo.lock

@junjihashimoto can you rebase ?

Sure!

I had success with the last version but now it breaks while building postgresql.

❯ nix develop .
warning: Git tree '/data/home/poelzi/Projects/sui/sui' is dirty
error: builder for '/nix/store/qq3g9i45ngsmc56a0d9dvq3bvinxv1v7-postgresql-16.4.drv' failed with exit code 2;
       last 10 log lines:
       > ok 214       + oidjoins                                 1020 ms
       > ok 215       - fast_default                              686 ms
       > ok 216       - tablespace                               1526 ms
       > 1..216
       > # 3 of 216 tests failed.
       > # The differences that caused some tests to fail can be viewed in the file "/build/postgresql-16.4/src/test/regress/regression.diffs".
       > # A copy of the test summary that you see above is saved in the file "/build/postgresql-16.4/src/test/regress/regression.out".
       > make[1]: *** [GNUmakefile:118: check] Error 1
       > make[1]: Leaving directory '/build/postgresql-16.4/src/test/regress'
       > make: *** [GNUmakefile:69: check] Error 2
       For full logs, run 'nix log /nix/store/qq3g9i45ngsmc56a0d9dvq3bvinxv1v7-postgresql-16.4.drv'.

Is there something we can help with here? Testing or anything

@TomaSajt
Thank you! The code fixes and the rebase are done.

@TomaSajt Please let me know if there's anything else blocking the approval or if you have any other concerns.

We should wait for the mass migration PRs to reach master, first.

After that, the packages that still use $cargoDepsCopy should be fixed manually (maybe in this PR).

@TomaSajt
I'd like to consider this using a concrete example. It is used in the following places, but which one is problematic?
Assuming all of them are problems, I'll try adding a "registry" directory to each one and compare the results.

$ git grep cargoDepsCopy
pkgs/applications/blockchains/zcash/default.nix:    # cargoDepsCopy gets unset after postPatch.
pkgs/applications/blockchains/zcash/default.nix:    configureFlagsArray+=("RUST_VENDORED_SOURCES=$cargoDepsCopy")
pkgs/applications/misc/pagefind/default.nix:      cd $cargoDepsCopy/lindera-unidic
pkgs/build-support/rust/hooks/cargo-setup-hook.sh:        export cargoDepsCopy="$(realpath "$(stripHash $cargoDeps)")"
pkgs/build-support/rust/hooks/cargo-setup-hook.sh:        cargoDepsCopy="$(realpath "$(pwd)/$sourceRoot/${cargoRoot:+$cargoRoot/}${cargoVendorDir}")"
pkgs/build-support/rust/hooks/cargo-setup-hook.sh:    config="$cargoDepsCopy/.cargo/config.toml"
pkgs/build-support/rust/hooks/cargo-setup-hook.sh:      --subst-var-by vendor "$cargoDepsCopy"
pkgs/build-support/rust/hooks/cargo-setup-hook.sh:    cargoDepsLockfile="$cargoDepsCopy/Cargo.lock"
pkgs/build-support/rust/hooks/cargo-setup-hook.sh:      echo "Cargo.lock is not the same in $cargoDepsCopy"
pkgs/build-support/rust/hooks/cargo-setup-hook.sh:    unset cargoDepsCopy
pkgs/by-name/ca/cargo-tauri/test-app.nix:    substituteInPlace $cargoDepsCopy/libappindicator-sys-*/src/lib.rs \
pkgs/by-name/ce/celeste/package.nix:    pushd $cargoDepsCopy/librclone-sys
pkgs/by-name/cl/clash-verge-rev/unwrapped.nix:    substituteInPlace $cargoDepsCopy/libappindicator-sys-*/src/lib.rs \
pkgs/by-name/cl/clash-verge-rev/unwrapped.nix:    substituteInPlace $cargoDepsCopy/sysproxy-*/src/linux.rs \
pkgs/by-name/di/difftastic/package.nix:    patch -d $cargoDepsCopy/libmimalloc-sys-0.1.24/c_src/mimalloc \
pkgs/by-name/fi/firecracker/package.nix:    substituteInPlace $cargoDepsCopy/aws-lc-sys-*/aws-lc/crypto/asn1/a_bitstr.c \
pkgs/by-name/gi/gitbutler/package.nix:    tomlq -ti 'del(.lints) | del(.workspace.lints)' "$cargoDepsCopy"/gix*/Cargo.toml
pkgs/by-name/mi/mistral-rs/package.nix:    rm "$cargoDepsCopy"/llguidance-*/build.rs
pkgs/by-name/os/oscavmgr/package.nix:    alvr_session=$(echo $cargoDepsCopy/alvr_session-*/)
pkgs/by-name/ov/overlayed/package.nix:    substituteInPlace $cargoDepsCopy/libappindicator-sys-*/src/lib.rs \
pkgs/by-name/po/pot/package.nix:    substituteInPlace $cargoDepsCopy/libappindicator-sys-*/src/lib.rs \
pkgs/by-name/rq/rquickshare/package.nix:    substituteInPlace $cargoDepsCopy/libappindicator-sys-*/src/lib.rs \
pkgs/by-name/ru/rustdesk-flutter/package.nix:    if [ $cargoDepsCopy ]; then # That will be inherited to buildDartPackage and it doesn't have cargoDepsCopy
pkgs/by-name/ru/rustdesk-flutter/package.nix:      substituteInPlace $cargoDepsCopy/libappindicator-sys-*/src/lib.rs \
pkgs/by-name/sl/slimevr/package.nix:      pushd $cargoDepsCopy/libappindicator-sys
pkgs/by-name/st/stardust-xr-server/package.nix:    install -D ${cpm-cmake}/share/cpm/CPM.cmake $(echo $cargoDepsCopy/stereokit-sys-*/StereoKit)/build/cpm/CPM_0.32.2.cmake
pkgs/by-name/te/termusic/package.nix:    pushd $cargoDepsCopy/stream-download
pkgs/by-name/te/termusic/package.nix:    substituteInPlace $cargoDepsCopy/stream-download/src/lib.rs \
pkgs/by-name/tr/treedome/package.nix:    substituteInPlace $cargoDepsCopy/libappindicator-sys-*/src/lib.rs \
pkgs/by-name/ve/veloren/package.nix:    cat <<'EOF' | tee "$cargoDepsCopy"/vek-*/build.rs
pkgs/development/tools/devpod/default.nix:        substituteInPlace $cargoDepsCopy/libappindicator-sys-*/src/lib.rs \
pkgs/servers/clickhouse/default.nix:    corrosionDepsCopy="$cargoDepsCopy"
pkgs/servers/clickhouse/default.nix:    rustDepsCopy="$cargoDepsCopy"
pkgs/servers/clickhouse/default.nix:      cargoDepsCopy="$corrosionDepsCopy" cargoSetupPostPatchHook
pkgs/servers/clickhouse/default.nix:      cargoDepsCopy="$rustDepsCopy" cargoSetupPostPatchHook

It might be better to add an option to make it compatible.

We really need to get this fix landed. Using the branch requires hours of compilation time and vendor is clearly broken with out.

I picked up the fetchCargoVendor part of the PR in #387337.

This way we can solve the issue quicker, without having to figure out what's the simplest and/or best implementation for importCargoLock.

I picked up the fetchCargoVendor part of the PR in #387337.

This way we can solve the issue quicker, without having to figure out what's the simplest and/or best implementation for importCargoLock.

Any updates on this?

FWIW: until these PRs land, you can just vendor the functions locally without needing to rebuild the world.

I’m going to read up on what happened here and split this PR into just the importCargoLock fixes, as @TomaSajt already split off the fetchCargoVendor ones. (@junjihashimoto Would you mind if I reused this PR/branch, or should I just open a new one?)

FWIW: until these PRs land, you can just vendor the functions locally without needing to rebuild the world.

I’m going to read up on what happened here and split this PR into just the importCargoLock fixes, as @TomaSajt already split off the fetchCargoVendor ones. (@junjihashimoto Would you mind if I reused this PR/branch, or should I just open a new one?)

Can you give a little hint what the best way would be to achieve this? I would have tried to apply an overlay patch...

@itpropro Let's not clutter this thread more -- join #rust:nixos.org or #users:nixos.org on Matrix and ping me, or post on Discourse and tag me.

@itpropro Let's not clutter this thread more -- join #rust:nixos.org or #users:nixos.org on Matrix and ping me.

I'm coming here to figure this out as well. Are you sure this quite important information should move whole other platform?

Asking on every PR "how do I apply this PR" is practically spam, ask on discourse.nixos.org if you want.

PR comments are for feedback on the content of the PR.

To be honest, I don't have the authority to merge, so it would be helpful if someone motivated could take over and accomplish the task.
We may be able to merge once all the packages are compatible.
However, it feels like the order is reversed, and I would like to merge first and move forward with the process of fixing the problems, but that's not the case, so it may be difficult to resolve.

Asking on every PR "how do I apply this PR" is practically spam, ask on discourse.nixos.org if you want.

PR comments are for feedback on the content of the PR.

To be fair, that was not what was asked, because just applying the PR didn't work in this case. It was especially about the part

FWIW: until these PRs land, you can just vendor the functions locally without needing to rebuild the world.
which is something that definitely needs explaining.

I agree that both of these should not be discussed here, but you have to consider that more and more packages are not updated because of the rust issues, so people will naturally stumble upon this PR when looking for a solution.

However, it feels like the order is reversed, and I would like to merge first and move forward with the process of fixing the problems

At the very least, if we need to revert later, reverting a treewide should be independent of the breaking change; we've seen issues in the past when trying to tackle both in the same PR. Merge conflicts also get much more onerous to handle.

I’m going to read up on what happened here and split this PR into just the importCargoLock fixes, as @TomaSajt already split off the fetchCargoVendor ones.

@winterqt based on my thoughts above: separate PR would be safer. (As long as they get credit for their work via Co-authored-by: or cherry-pick or so on)

based on my thoughts above: separate PR would be safer.

@eclairevoyant I’m not sure what you mean — to clarify, by “split this PR” I meant just removing the fetchCargoVendor changes from here, I didn’t ever mention doing a treewide. (Unless you were talking about an earlier comment of yours, not the treewide part?)

There are packages that relied on the layout of their vendor directory, those needed a treewide update, because our changes would break that assumption.

In #387337 I have a treewide fix for packages that use fetchCargoVendor.
IIRC there are only 3 packages that still use importCargoLock that rely on the layout, I also listed them in the other PR.

Of course, out-of-tree packages making this assumption will break no matter what, but they'll just have to deal with it...

So yeah, a "treewide" (3 packages) commit is still missing from this PR (or its split off continuation).

Yeah, so let’s include the fixes here — no need for a separate PR. My question was about whether I should reuse/push to this PR, or if I should open a new one.

Pushing here sounds fine then, if you have access to do so.