guix: Add guix-{attest,verify} scripts #21462

dongcarl · 2021-03-17T20:22:28Z

Adds replacements for gsign and gverify.

Personally I'm not a big fan of using the word "sign" as it's been used to refer to both codesigning and GPG signing.

DrahtBot · 2021-03-17T20:46:31Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

dongcarl · 2021-04-09T01:36:01Z

Perhaps we can test this by doing a trial run? I've PR'ed my attestations of 5da0e36 here: bitcoin-core/guix.sigs#1

What I did:

Run guix-build
Run env GUIX_SIGS_REPO=<path/to/guix.sigs> SIGNER=0x96AB007F1A7ED999=dongcarl ./contrib/guix/guix-attest

Once you've run ./contrib/guix/guix-attest, you can verify against my signatures using:

$ env GUIX_SIGS_REPO=<path/to/guix.sigs> ./contrib/guix/guix-verify

hebasto

Tested 5da0e36.

Result: bitcoin-core/guix.sigs#2

Once you've run ./contrib/guix/guix-attest, you can verify against my signatures using:
$ env GUIX_SIGS_REPO=<path/to/guix.sigs> ./contrib/guix/guix-verify

I think it is possible when bitcoin-core/guix.sigs#1 is merged, no?

fanquake · 2021-04-10T01:00:17Z

I think it is possible when bitcoin-core/guix.sigs#1 is merged, no?

The PR doesn't have to be merged, you should be able to just fetch and checkout the branch you want to verify against.

fanquake · 2021-04-10T03:10:25Z

I have done a trial run through the attestation / verify steps as well. bitcoin-core/guix.sigs#3.

Note that for anyone attesting/verifying on macOS, the invocations of find and xargs used in the attestation script are not currently compatible with BSD find or xargs. I just swapped them out for gfind and gxargs for now.

sipa · 2021-04-10T19:20:29Z

contrib/guix/guix-attest

+            find . -type f -printf '%P\0' | env LC_ALL=C sort -z | xargs -r0 sha256sum >> "$outsigdir"/SHA256SUMS
+        )
+        echo "${outname}: Signing SHA256SUMS to produce SHA256SUMS.asc"
+        gpg --detach-sign --local-user "$gpg_key_name" --output "$outsigdir"/SHA256SUMS.asc "$outsigdir"/SHA256SUMS


If you're producing .asc files I think it would be expected that they're ascii-armored (--armor); otherwise use the .sig extension (this doesn't really matter, but it's a convention).

RonSherfey

requested change

dongcarl · 2021-04-20T20:33:48Z

Pushed 816c70c -> 7cd57f3

The build output is now constructed in $DISTSRC/output, then moved (likely atomically) to $OUTDIR when everything is done
Inputs are now attested to, thanks to sipa's suggestions

Please see commit messages for more explanations.

Signatures up here: https://github.com/dongcarl/guix.sigs/tree/2021-04-PR21462/7cd57f3ac308

dongcarl · 2021-04-28T15:55:46Z

Wondering if the latest changes look alright, and what else needs fixing up here before it's ready for merge!

maflcko

Concept ACK 7cd57f3 🍠

Show signature and timestamp

Signature:

Timestamp of file with hash bc7b8ef6e6110dba1bb1148d42a8626cc8670590bd75fcef192668f241327364 -

maflcko · 2021-04-30T09:49:59Z

contrib/guix/guix-attest

+echo ""
+
+# MAIN LOGIC: Loop through each output for VERSION and attest to output in
+#             GUIX_SIGS_REPO as SIGNERif attestation does not exist


8dd3c4a:

Suggested change

# GUIX_SIGS_REPO as SIGNERif attestation does not exist

# GUIX_SIGS_REPO as SIGNER, if attestation does not exist

maflcko · 2021-04-30T10:17:08Z

contrib/guix/guix-attest

+            if [ -e inputs.SHA256SUMS ]; then
+                echo "${outname}: Including existent input SHA256SUMS"
+                cat inputs.SHA256SUMS >> "$outsigdir"/SHA256SUMS
+            fi


shouldn't this fail when the file is missing?

Thinking about this since dist-archive does not have an inputs.SHA256SUMS... Right now, we attest to the bitcoin repo's git-archive in distsrc-archive, but I think perhaps it only makes sense to attest to it in the context of it being inputs to the host-specific outputs. When we add codesignatures, similarly it only makes sense to attest to the detached-sigs repo's git-archive in the context of it being inputs to the host-specific codesigned outputs. Will make a change so that we don't attest to dist-archive directly.

DrahtBot · 2021-05-03T09:33:18Z

🕵️ @practicalswift has been requested to review this pull request as specified in the REVIEWERS file.

While files are being output to $OUTDIR, it will be under ${DISTSRC}/output, and only when everything is done, will ${DISTSRC}/output be moved to the actual $OUTDIR. This makes it so that a Ctrl-C in the middle of a build is less likely to result in a partially-constructed $OUTDIR. In fact, if I understand correctly, if $OUTDIR and $DISTSRC reside on the same filesystem, the move (rename) is likely atomic. Also, since the "working $OUTDIR" is under ${DISTSRC}/output, it will be cleaned properly by the guix-clean script.

At build/codesigning-time, hash build inputs and output the digest to ${OUTDIR}/inputs.SHA256SUMS, which gets included in the final SHA256SUMS constructed by guix-attest. Example final SHA256SUMS: ee832d2a35b7701bff581dea05a536118b118e3ad0a587a2855b6ee8cd6fba20 inputs/bitcoin-78199266af7b.tar.gz ca765e70a0c12866dd63c0be228b675278a26329e5f8f5b5c52fd09200fedf21 bitcoin-78199266af7b-powerpc64le-linux-gnu-debug.tar.gz dae95327d7f2c324e2728c4b73627be6cb2c0d2f2e5bea940d1d5e6463939327 bitcoin-78199266af7b-powerpc64le-linux-gnu.tar.gz

We already attest to the relevant dist-archive in inputs.SHA256SUMS, which is recorded at build-time. We use a SKIPATTEST.TAG file to indicate output directories which do not require attestation (much like the CACHEDIR.TAG specification). Generally, it's better to have build scripts declare properties of directories instead of introducing name-based special cases in attest scripts since build scripts have a more detailed context of what is going on.

dongcarl · 2021-05-03T18:50:23Z

Pushed 7cd57f3 -> feda2c8

Addressed: guix: Add guix-{attest,verify} scripts #21462 (comment), guix: Add guix-{attest,verify} scripts #21462 (comment) (see last commit)

As always, guix.sigs are up at: https://github.com/dongcarl/guix.sigs/tree/2021-04-PR21462/feda2c8e3180

Sometimes GPG connects to the wrong agent... or you don't have your smartcard handy...

dongcarl · 2021-05-04T15:56:41Z

Pushed feda2c8..d420e5c:

Added an ERR trap to remove incomplete sigdirs if signing fails

guix.sigs up at: https://github.com/dongcarl/guix.sigs/tree/2021-04-PR21462/d420e5c1c015

laanwj · 2021-05-05T20:09:04Z

Tested ACK
I get the same output.

for x in *; do diff -du $x/dongcarl/SHA256SUMS $x/laanwj/SHA256SUMS; done

Would it be useful to have a concatenated SHA256SUMS (with all files) as well? I mean, depending on how we're going to do the distribution on the website and torrent.

I guess if we're redesigning this it makes sense to have the two match up, not sign a different file with the 'distribution signing key' anymore but the same one (or forego my distribution signing key completely, just collect the attestations).

Or is the idea is to start using a subdirectory per platform there (each with its own SHA256SUMS) too? In this case, having inputs/bitcoin-d420e5c1c015.tar.gz in every one might not be practical, it implies having to duplicate the tarball . Maybe ../bitcoin-d420e5c1c015.tar.gz then have the tarball in the top-level directory?

dongcarl · 2021-05-07T19:41:39Z

Or is the idea is to start using a subdirectory per platform there (each with its own SHA256SUMS) too? In this case, having inputs/bitcoin-d420e5c1c015.tar.gz in every one might not be practical, it implies having to duplicate the tarball .

Hmmm, are you talking about for the webserver? I think it can be done using hard/symlinks at the FS level, or just using virtualized paths at the HTTP level.

Wondering what particular user-flow you have in mind :-)

laanwj · 2021-05-12T11:20:31Z

Code review and tested ACK d420e5c

Wondering what particular user-flow you have in mind :-)

After discussion on IRC: let's leave this for another PR.

d420e5c guix-attest: Avoid incomplete sigdirs with ERR traps (Carl Dong) feda2c8 guix: Skip attesting to dist-archive (Carl Dong) d522d80 guix: Attest to inputs in inputs.SHA256SUMS (Carl Dong) f9e2960 guix: Construct $OUTDIR in ${DISTSRC}/output (Carl Dong) 022abc8 guix: Minor quoting fix in libexec/build.sh (Carl Dong) c83c4fa guix-attest: Allow skipping GPG signing with NO_SIGN (Carl Dong) 0e1c2e4 guix-attest: Use ascii-armor signatures (Carl Dong) b5fd89c guix-attest: Only use cross-platform flags for find+xargs (Carl Dong) 5926432 guix: Add guix-verify script (Carl Dong) 30daf76 guix: Add guix-attest script (Carl Dong) Pull request description: Adds replacements for `gsign` and `gverify`. Personally I'm not a big fan of using the word "sign" as it's been used to refer to both codesigning and GPG signing. ACKs for top commit: laanwj: Code review and tested ACK d420e5c Tree-SHA512: 93d82d201f4596eaea0e3825aa55b013dfb91790e6ccee79893833d37921513d7b4e735f0641103e1e2ea8308abe4cb6218b73160924708802f2e0e3f7f6caf1

Sjors · 2021-05-14T08:43:25Z

contrib/guix/guix-attest

+
+            if [ -e inputs.SHA256SUMS ]; then
+                echo "${outname}: Including existent input SHA256SUMS"
+                cat inputs.SHA256SUMS >> "$outsigdir"/SHA256SUMS


This line fails with "path not found" on macOS when GUIX_SIGS_REPO is a relative path like ../guix.sigs.

Sjors · 2021-05-14T08:58:31Z

contrib/guix/guix-attest

+            echo "${outname}: Hashing build outputs to produce SHA256SUMS"
+            files="$(find -L . -type f ! -iname '*.SHA256SUMS')"
+            if [ -n "$files" ]; then
+                cut -c3- <<< "$files" | env LC_ALL=C sort | xargs sha256sum >> "$outsigdir"/SHA256SUMS


On macOS this needs to be shasum -a 256 or the user has to make an alias or brew install coreutils

Hmmm, does shasum -a 256 produce identical output to sha256sum?

If my uploaded attestation is correct, yes :-)

fac4814 doc/release-process: Add torrent creation details (Carl Dong) 5d24cc3 guix/INSTALL: Guix installs init scripts in libdir (Carl Dong) 5da2ee4 guix/INSTALL: Add coreutils/inotify-dir-recreate troubleshooting (Carl Dong) 318c607 guix: Adapt release-process.md to new Guix process (Carl Dong) fcab35b guix-attest: Produce and sign normalized documents (Carl Dong) c2541fd guix: Overhaul README (Carl Dong) 46ce6ce tree-wide: Rename gitian-keys to builder-keys (Carl Dong) fc4f844 guix: Update various check_tools lists (Carl Dong) 263220a guix: Check for a sane services database (Carl Dong) Pull request description: Based on: #21462 Keeping the README in one file so that it's easy to search through. Will add more jumping links later so navigation is easier. Current TODOs: - [x] Shell installer option: prompt user to re-login for `/etc/profile.d` entry to be picked up - [x] Binary tarball option: prompt user to create `/etc/profile.d` entry and re-login - [x] Fanquake docker option: complete section - [x] Arch Linux AUR option: prompt to start `guix-daemon-latest` unit after finishing "optional setup" section - [x] Building from source option: Insert dependency tree diagram that I made - [x] Building from source option: redo sectioning, kind of a mess right now - [x] Optional setup: make clear which parts are only needed if building from source - [x] Workaround 1 for GnuTLS: perhaps mention how to remove Guix build farm's key - [x] Overall (after everything): Make the links work. Note to self: wherever possible, tell user how to check that something is true rather than branching by installation option. ACKs for top commit: fanquake: ACK fac4814 - going to go ahead and merge this now. It's a lot of documentation, and could probably be nit-picked / improved further, however, that can continue over the next few weeks. I'm sure more (backportable) improvements / clarifications will be made while we progress through RCs towards a new release. Tree-SHA512: dc46c0ecdfc67c7c7743ca26e4a603eb3f54adbf81be2f4c1f4c20577ebb84b5250b9c9ec89c0e9860337ab1c7cff94d7963c603287267deecfe1cd987fa070a

maflcko added this to the 22.0 milestone Mar 17, 2021

DrahtBot added Build system Docs Scripts and tools labels Mar 17, 2021

This was referenced Mar 17, 2021

guix: Do not treat directory existence as an error #21429

Closed

build, qt: No longer need to patch translation.pro #21420

Closed

This was referenced Mar 17, 2021

Guix Release Transition Plan for 22.0 #21145

Closed

guix: Brainstorm w/re guix.sigs repo layout #21273

Closed

DrahtBot mentioned this pull request Mar 18, 2021

build: Bump clang version to fix non-determinism #20454

Closed

maflcko removed Docs Build system Scripts and tools labels Mar 18, 2021

DrahtBot added Build system Scripts and tools labels Mar 18, 2021

This was referenced Mar 18, 2021

build: macOS toolchain bump #19817

Merged

build, qt: Fix static builds on macOS Big Sur #21495

Merged

Qt: Add Android packaging support #17227

Merged

dongcarl force-pushed the 2021-03-guix-verify-and-attest branch 2 times, most recently from e896333 to 5da0e36 Compare April 9, 2021 00:28

dongcarl mentioned this pull request Apr 9, 2021

Add 5da0e361b8b5 non-codesigned signatures bitcoin-core/guix.sigs#1

Closed

dongcarl marked this pull request as ready for review April 9, 2021 01:33

hebasto mentioned this pull request Apr 9, 2021

Add hebasto 5da0e361b8b5 non-codesigned signatures bitcoin-core/guix.sigs#2

Closed

hebasto reviewed Apr 9, 2021

View reviewed changes

sipa reviewed Apr 10, 2021

View reviewed changes

RonSherfey suggested changes Apr 11, 2021

View reviewed changes

RonSherfey approved these changes Apr 11, 2021

View reviewed changes

dongcarl mentioned this pull request Apr 16, 2021

guix: Add full installation and usage documentation #21711

Merged

9 tasks

RonSherfey approved these changes Apr 21, 2021

View reviewed changes

maflcko reviewed Apr 30, 2021

View reviewed changes

dongcarl added 9 commits May 3, 2021 13:18

guix: Add guix-attest script

30daf76

guix: Add guix-verify script

5926432

guix-attest: Only use cross-platform flags for find+xargs

b5fd89c

guix-attest: Use ascii-armor signatures

0e1c2e4

guix-attest: Allow skipping GPG signing with NO_SIGN

c83c4fa

guix: Minor quoting fix in libexec/build.sh

022abc8

dongcarl force-pushed the 2021-03-guix-verify-and-attest branch from 7cd57f3 to feda2c8 Compare May 3, 2021 17:18

dongcarl mentioned this pull request May 3, 2021

guix: Use gcc-8 across the board #21799

Merged

guix-attest: Avoid incomplete sigdirs with ERR traps

d420e5c

Sometimes GPG connects to the wrong agent... or you don't have your smartcard handy...

laanwj merged commit 01624a7 into bitcoin:master May 12, 2021

Sjors reviewed May 14, 2021

View reviewed changes

hebasto mentioned this pull request May 24, 2021

Add hebasto Guix attestations for ee883201cf13 win-codesigned bitcoin-core/guix.sigs#12

Closed

gwillen pushed a commit to ElementsProject/elements that referenced this pull request Jun 1, 2022

Merge 01624a7 into merged_master (Bitcoin PR bitcoin/bitcoin#21462)

1207874

bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022

	# GUIX_SIGS_REPO as SIGNERif attestation does not exist
	# GUIX_SIGS_REPO as SIGNER, if attestation does not exist

guix: Add guix-{attest,verify} scripts #21462

guix: Add guix-{attest,verify} scripts #21462

Uh oh!

Conversation

dongcarl commented Mar 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Mar 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Conflicts

Uh oh!

dongcarl commented Apr 9, 2021

Uh oh!

hebasto left a comment

Choose a reason for hiding this comment

Uh oh!

fanquake commented Apr 10, 2021

Uh oh!

fanquake commented Apr 10, 2021

Uh oh!

sipa Apr 10, 2021

Choose a reason for hiding this comment

Uh oh!

RonSherfey left a comment

Choose a reason for hiding this comment

Uh oh!

dongcarl commented Apr 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongcarl commented Apr 28, 2021

Uh oh!

maflcko left a comment

Choose a reason for hiding this comment

Uh oh!

maflcko Apr 30, 2021

Choose a reason for hiding this comment

Uh oh!

maflcko Apr 30, 2021

Choose a reason for hiding this comment

Uh oh!

dongcarl Apr 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DrahtBot commented May 3, 2021

Uh oh!

dongcarl commented May 3, 2021

Uh oh!

dongcarl commented May 4, 2021

Uh oh!

laanwj commented May 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongcarl commented May 7, 2021

Uh oh!

laanwj commented May 12, 2021

Uh oh!

Sjors May 14, 2021

Choose a reason for hiding this comment

Uh oh!

Sjors May 14, 2021

Choose a reason for hiding this comment

Uh oh!

dongcarl May 14, 2021

Choose a reason for hiding this comment

Uh oh!

Sjors May 14, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dongcarl commented Mar 17, 2021 •

edited

Loading

DrahtBot commented Mar 17, 2021 •

edited

Loading

dongcarl commented Apr 20, 2021 •

edited

Loading

dongcarl Apr 30, 2021 •

edited

Loading

laanwj commented May 5, 2021 •

edited

Loading