I recently switched to a new GUIX build setup...

Mind providing its description or steps to reproduce it?

I've recently started using the NixOS GUIX module, which might be doing something slightly different resulting in different hashes.

I've re-triggered a 27.0 build with this setup to then compare binaries.

Might be related to bitcoin/bitcoin#29754, which implies not 100% isolated build environment.

I recently switched to a new GUIX build setup...

Mind providing its description or steps to reproduce it?

On a system running NixOS, enable the GUIX service in configuration.nix with

services.guix = {
    enable = true;
};

then, make a GUIX build with contrib/guix/guix-build.

Looking at the 27.0 output, curiously, these two files are the same as the others:

 9c1ee651d3b157baccc3388be28b8cf3bfcefcd2493b943725ad6040ca6b146b  bitcoin-27.0.tar.gz
 e8114ed85a976ff439bd78cbf026e3f9bfafdf40d0fe75121e73bd4b7af347a4  bitcoin-27.0-win64-setup-unsigned.exe

The others all diverge.

i can understand about the source tarball, but that win64 also matches (but not any of the others) is... strange. It makes me think the source of the divergence is some filesystem metadata-related (the tarball is generated by git(?), and otherwise only the .exe won't have any extended UNIX file attributes).

I've re-triggered a 27.0 build with this setup to then compare binaries.

Would imagine they all differ in a similar way. so for comarison it'd be enough to upload say, x86_64 linux and maybe the associated -debug (if we need to go that deep to compare symbol addresses).

edit: i've installed guix on a nixos VM, let's try to reproduce !

i've been able to reproduce @0xB10C's results:

[nix-shell:/data/build/bitcoin/guix-build-27.0/output/x86_64-linux-gnu]$ sha256sum *
30bcdcff385ce2f790613e499a2cde0b66aee082c1053bdc619723a553601d87  bitcoin-27.0-x86_64-linux-gnu-debug.tar.gz
9ddf93c4548cb0fa14a687dc131685eb54611a7e4d99626cc4f319b54f3cfc46  bitcoin-27.0-x86_64-linux-gnu.tar.gz

Will compare them to the release.

Edit:

• Contents of the archives seem to be exactly the same. So this is a difference on the tar or gz level.
• Decompressed tar files are the same size, yet have different SHA256.
• The only difference is indeed in the metadata, as i expected. This for every file:

--- release/bitcoin-27.0-x86_64-linux-gnu.tar.hex       2024-04-22 19:54:17.357526815 +0200                            
+++ nix/bitcoin-27.0-x86_64-linux-gnu.tar.hex   2024-04-22 19:53:26.654081476 +0200                                    
@@ -3,8 +3,8 @@                                                                                                        
 *                                                                                                                     
 00000060  00 00 00 00 30 30 30 30  37 35 35 00 30 30 30 30  |....0000755.0000|                                        
 00000070  30 30 30 00 30 30 30 30  30 30 30 00 30 30 30 30  |000.0000000.0000|                                        
-00000080  30 30 30 30 30 30 30 00  31 34 36 30 37 32 35 32  |0000000.14607252|                                        
-00000090  36 36 30 00 30 30 37 37  36 37 00 20 35 00 00 00  |660.007767. 5...|                                        
+00000080  30 30 30 30 30 30 30 00  30 32 32 36 33 35 32 33  |0000000.02263523|                                        
+00000090  30 30 30 00 30 30 37 37  34 37 00 20 35 00 00 00  |000.007747. 5...|                                        
 000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|                                        
 *                                                                                                                     
 00000100  00 75 73 74 61 72 20 20  00 00 00 00 00 00 00 00  |.ustar  ........|                                        

• Comparing tar -tvf output, it's the date! Release:

 drwxr-xr-x 0/0               0 2024-04-15 18:28 bitcoin-27.0/ 

Nix:

 drwxr-xr-x 0/0               0 1980-01-01 01:00 bitcoin-27.0/ 

Does Nix use a patched tar, maybe? But the same is true for .zips, apparently.

No, it doesn't. But nix builds in a sandbox that sets all dates to 1980-01-01. This is what we're seeing here.

In contrib/guix/libexec/build.sh we do the following to deterministically set the dates for created archives to a specific epoch

For tar, pass --mtime option through the default options environment variable TAR_OPTIONS:

export TAR_OPTIONS="--owner=0 --group=0 --numeric-owner --mtime='@${SOURCE_DATE_EPOCH}' --sort=name"                                                                                                                                       

For zip (windows), use touch to set the date before compressing using zip:

                find "${DISTNAME}" -not -name "*.dbg" -print0 \
                    | xargs -0r touch --no-dereference --date="@${SOURCE_DATE_EPOCH}"

Both don't work inside Nix Guix, somehow they get squashed to 1980.

@0xB10C i think i solved it, just do

unset SOURCE_DATE_EPOCH
contrib/guix/guix-build

(we likely want to do this inside the build script, but this will work for now)

Wow @laanwj good detective work! I can confirm that the files in the tar.gz archives are set to 1980-01-01 and hash to the same (incorrect) hash. I've unset SOURCE_DATE_EPOCH and am doing another 27.0 GUIX build now.

Does it make sense to have a CI job that errors in cases where newly added hashes don't match the existing hashes for the same release?

Wow @laanwj good detective work!

Thank you !

Does it make sense to have a CI job that errors in cases where newly added hashes don't match the existing hashes for the same release?

We've discussed this in the past and although it's useful for detecting flukes and misconfigurations early, broadly i'm against this: i think there needs to be evidence of diverging hashes in the repository (even if removed or replaced in later commits, like here). After all, in an actual compromise, it's not 100% clear who has the correct output, it's not necessarily who submitted them first.

What i really like to avoid is anyone doing some crude hacks to get the same output just to "pass the CI" and have it merged (and introducing say, the backdoor), instead of investigating the difference.

We do need some kind of warning system though if there are diverging outputs in the first place. If they go undetected it's also pointless 😄

But it shouldn't ideally prevent merging.

We do need some kind of warning system though if there are diverging outputs in the first place. If they go undetected it's also pointless 😄

But it shouldn't ideally prevent merging.

A nightly CI job on main that opens an issue if there's a mismatch?

A nightly CI job on main that opens an issue if there's a mismatch?

That could work! In any case, it'd make sense to open a new issue in this repository for discussing this.