Skip to content

Conversation

achow101
Copy link
Member

@achow101 achow101 commented Aug 6, 2021

The SHA256SUMS file can be used in a sha256sum -c command to verify downloaded binaries. However users are likely to download just a single file and not place this file in the correct directory relative to the SHA256SUMS file for the simple verification command to work. By not including the directory name in the SHA256SUMS file, it will be easier for users to verify downloaded binaries.

@achow101 achow101 force-pushed the shasum-no-dir branch 2 times, most recently from c1f2435 to 2dabd97 Compare August 6, 2021 22:36
@ghost
Copy link

ghost commented Aug 7, 2021

Why don't you mention this PR in #22634 (comment) ?

I have now rewritten my download&verfiy bash script already. It's true that I e.g. do not download into a subfolder "x86_64-linux-gnu", so as you said, I experienced that sha256sum -c --ignore-missing does not work.

My workaround was just to do grep <Tarballname> SHA256SUMS and then sha256sum <Tarballname> to be able to do manual visual comparison of the two output hashes (which I did also with the previous SHA256SUM format file, which contained the signature itself).

I am not sure if this is good or bad, on one hand I like it that the SHA256SUMS file mirrors the download directory structure, that is clean and I think it would not be feasible to have an SHA256SUMS file in every arch sub dir, because that would require also a signature file for each of them. On the other hand, I agree, that normally one does not download in an arch sub dir. But the latter could be done easily I think, though.

@hebasto
Copy link
Member

hebasto commented Aug 7, 2021

Concept ACK. Arch is included in the output filenames, therefore, no name clash could be expected.

@achow101 achow101 mentioned this pull request Aug 7, 2021
Copy link
Member

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 2dabd97, I have reviewed the code and it looks OK, I agree it can be merged.

Copy link
Contributor

@Zero-1729 Zero-1729 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 2dabd97

LGTM, although, I would suggest adding the comment proposed here #22654 (comment) to make future code review easier.

Copy link
Member

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK 568ef8d, only a comment added since my previous review.

Copy link
Contributor

@Zero-1729 Zero-1729 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK 568ef8d

Nice comment addition.

Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see my comment

Sorry that this is now in the "Conversation" section, I only tried out if one could some kind of "reopen" the conversation in the "Files changed" tab, which was marked as resolved by Achow, instead of only adding a comment to it. Now I cannot find out how to remove this.

@Zero-1729
Copy link
Contributor

Zero-1729 commented Aug 9, 2021

Great find @wodry!

EDIT: Given using $0 as a bash param is discouraged (see here), this change is good to go.

@ghost
Copy link

ghost commented Aug 9, 2021

Concept ACK. Arch is included in the output filenames, therefore, no name clash could be expected.

Just asking, why arch subfolders at all, if every 22.0rc2 subfolder contains just one file with a unique name which contains the arch, except the folder x86_64-w64-mingw32/ which has two files in it, also without clash?

@laanwj
Copy link
Member

laanwj commented Aug 10, 2021

As I understand this makes the uploaded binaries have all files in one directory like previous releases? (e.g. https://bitcoincore.org/bin/bitcoin-core-0.21.1/ instead of https://bitcoincore.org/bin/bitcoin-core-22.0/test.rc2/)

I didn't think nesting the directory per platform makes anything easier or better, but does add an inconvenient extra click when downloading. So concept ACK.

I have now rewritten my download&verfiy bash script already.

I would suggest holding off doing any large tool rewrites based on process changes until 22.0 final. Things are still being hammered out.

@laanwj laanwj added this to the 22.0 milestone Aug 10, 2021
Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code change ACK, Concept 0

@ghost
Copy link

ghost commented Aug 10, 2021

As I understand this makes the uploaded binaries have all files in one directory like previous releases?

I am not sure what you are referring to with "this"?
This PR leaves the arch sub dir structure as is, only removes the directories from the SHA256SUMS file.
Not sure what and why you ACK.

@laanwj
Copy link
Member

laanwj commented Aug 10, 2021

Not sure what and why you ACK.

Having a single-level SHA256SUMS means that we can put the files in one directory again during upload (anyhow, that users are able to verify that without having to move files around or pre-process SHA256SUMS).

@achow101
Copy link
Member Author

achow101 commented Aug 10, 2021

As I understand this makes the uploaded binaries have all files in one directory like previous releases?

Unless the binary server validates the SHA256SUMS file, I don't think so. I don't think it really matters where the binaries are located. Regardless of the directory structure, users are still going to just download one binary file and the SHA256SUMS file into some Downloads directory and verify them when they are at the same level.

@achow101
Copy link
Member Author

From IRC:

<dongcarl> The layout of the SHA256SUMS file is designed such that anyone can download any subtree (as torrent download tools allow you to do), then `cd path/to/all/output/basedir; sha256sum --check --ignore-missing` to verify their files. In that sense it needs to represent represent the tree somewhat accurately.

So I guess the directory structure will need to change for the upload. I will add something to doc/release-process.md to reflect this.

@achow101 achow101 force-pushed the shasum-no-dir branch 2 times, most recently from c602555 to 56c25f5 Compare August 10, 2021 21:07
} | xargs realpath --relative-base="$PWD" \
| xargs sha256sum \
# Get all of the generated output files and hash them without directory names.
# Because "find" prints filenames with a preceding "./", we need to pass the filenames into "basename" first.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally I think it would be good to leave the preceding ./ path, because it is precise.
The verifying works with it on Linux, so the removing is not generally needed.
I guess it shall be removed to be compatible with OSs that have other directory naming conventions, e.g. like Microsoft products?
I would find a comment helpful, for what reason the ./ is needed to be removed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typically SHA256SUMS do not contain paths to the files. Seeing the ./ may be confusing to users who are not familiar with paths.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW here (leading to the doc of md5sum --check invocation) is the doc of SHA256SUMS file:
The input to this mode of md5sum is usually the output of a prior, checksum-generating run of ‘md5sum’.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but many people do a visual comparison.

@dongcarl
Copy link
Contributor

dongcarl commented Aug 11, 2021

Here's an alternative to 752235f, which does the "basenamification" at attest-time (which I think makes a bit more sense):

diff --git a/contrib/guix/guix-attest b/contrib/guix/guix-attest
index 1503c330b2..2e2ac0b411 100755
--- a/contrib/guix/guix-attest
+++ b/contrib/guix/guix-attest
@@ -159,6 +159,18 @@ Hint: You may wish to remove the existing attestations and their signatures by
 EOF
 }
 
+# Give a SHA256SUMS file as stdin that has lines like:
+#     0ba536819b221a91d3d42e978be016aac918f40984754d74058aa0c921cd3ea6  a/b/d/c/d/s/bitcoin-22.0rc2-riscv64-linux-gnu.tar.gz
+#     ...
+#
+# Replace each line's file name with its basename:
+#     0ba536819b221a91d3d42e978be016aac918f40984754d74058aa0c921cd3ea6  bitcoin-22.0rc2-riscv64-linux-gnu.tar.gz
+#     ...
+#
+basenameify_SHA256SUMS() {
+    sed -E 's@(^[[:xdigit:]]{64}[[:space:]]+).+/([^/]+$)@\1\2@'
+}
+
 echo "Attesting to build outputs for version: '${VERSION}'"
 echo ""
 
@@ -174,6 +186,7 @@ mkdir -p "$outsigdir"
         cat "${noncodesigned_fragments[@]}" \
             | sort -u \
             | sort -k2 \
+            | basenameify_SHA256SUMS \
                 > "$temp_noncodesigned"
         if [ -e noncodesigned.SHA256SUMS ]; then
             # The SHA256SUMS already exists, make sure it's exactly what we
@@ -201,6 +214,7 @@ mkdir -p "$outsigdir"
         cat "${sha256sum_fragments[@]}" \
             | sort -u \
             | sort -k2 \
+            | basenameify_SHA256SUMS \
                 > "$temp_all"
         if [ -e all.SHA256SUMS ]; then
             # The SHA256SUMS already exists, make sure it's exactly what we

@fanquake
Copy link
Member

Yes I think so.

The SHA256SUMS file can be used in a sha256sum -c command to verify
downloaded binaries. However users are likely to download just a single
file and not place this file in the correct directory relative to the
SHA256SUMS file for the simple verification command to work. By not
including the directory name in the SHA256SUMS file, it will be easier
for users to verify downloaded binaries.

Co-authored-by: Carl Dong <contact@carldong.me>
@achow101
Copy link
Member Author

Changed to use @dongcarl's suggestion

The uploaded binaries need to match the same flat directory structure of
the SHA256SUMS file in order for torrent downloaders to be able to
verify the download without moving files. Mention this in the release
process doc.
Copy link
Contributor

@Zero-1729 Zero-1729 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK 132cae4

@fanquake
Copy link
Member

Guix builds:

777a1ad25da672e72356d34638fef917649462307e6a857e0c2f4d78b8d3e363  guix-build-132cae44f2d0/output/aarch64-linux-gnu/SHA256SUMS.part
5f307edda9d8785a7051bdc89e667500ddd932324f6a0b225d605a921236cdf5  guix-build-132cae44f2d0/output/aarch64-linux-gnu/bitcoin-132cae44f2d0-aarch64-linux-gnu-debug.tar.gz
99c0fe0aa4952af02384ebd4b84c490351ad462290e4673af5c16d13929d929a  guix-build-132cae44f2d0/output/aarch64-linux-gnu/bitcoin-132cae44f2d0-aarch64-linux-gnu.tar.gz
d30698cf2358edd729c9cae72270df510d2d811debbac1fd2619ad5aeed8eab3  guix-build-132cae44f2d0/output/arm-linux-gnueabihf/SHA256SUMS.part
2a39d1e11e63a1662d644e16d8002ebc778acaac04ac23572666fb57f15f1d07  guix-build-132cae44f2d0/output/arm-linux-gnueabihf/bitcoin-132cae44f2d0-arm-linux-gnueabihf-debug.tar.gz
807494da57ef85914988500f3533257c5ae09b9351ad742ab317cb4326a72fed  guix-build-132cae44f2d0/output/arm-linux-gnueabihf/bitcoin-132cae44f2d0-arm-linux-gnueabihf.tar.gz
31067da3649769233be6ccf8e0aebf0e09af2d70ae649c42011557b111fd0811  guix-build-132cae44f2d0/output/dist-archive/bitcoin-132cae44f2d0.tar.gz
a9b3aad4050e69ff5bc792c2057934da227e793a7ccd73514031bd8c4b48623c  guix-build-132cae44f2d0/output/powerpc64-linux-gnu/SHA256SUMS.part
23a0874ab176e0574351ff60e8a193196b48eb640b0a6f77188d7ceab5f70a3c  guix-build-132cae44f2d0/output/powerpc64-linux-gnu/bitcoin-132cae44f2d0-powerpc64-linux-gnu-debug.tar.gz
bd0d1ac1ffe179ba1d60259cdf718b24f72490c95656842bd2df0fc8f44e794d  guix-build-132cae44f2d0/output/powerpc64-linux-gnu/bitcoin-132cae44f2d0-powerpc64-linux-gnu.tar.gz
ab4d8038e2c1372d2beaaa5c647ba3b30cfe2e0afb38c2174f1264955dacae25  guix-build-132cae44f2d0/output/powerpc64le-linux-gnu/SHA256SUMS.part
41b493490edf41371fe39d6aaef392bec4f1531e26c746a989729cc1bf54f1a4  guix-build-132cae44f2d0/output/powerpc64le-linux-gnu/bitcoin-132cae44f2d0-powerpc64le-linux-gnu-debug.tar.gz
e6b79baa7ed8410f3d5cb4b2d21278c06ce9c665fd7ffe19c1eb3d2e1c147d91  guix-build-132cae44f2d0/output/powerpc64le-linux-gnu/bitcoin-132cae44f2d0-powerpc64le-linux-gnu.tar.gz
7e5ab13ac196511d14a5cb88272ce377257f0b117e5b69a9d8f64aaab8aea580  guix-build-132cae44f2d0/output/riscv64-linux-gnu/SHA256SUMS.part
155146bca7dbe8b499e8ff22395ea71fd770853e075e63293693e6718b1d15b2  guix-build-132cae44f2d0/output/riscv64-linux-gnu/bitcoin-132cae44f2d0-riscv64-linux-gnu-debug.tar.gz
de943b63789da014f642417800197988350be13bdaec2e6ac4dcd8dea0e26b85  guix-build-132cae44f2d0/output/riscv64-linux-gnu/bitcoin-132cae44f2d0-riscv64-linux-gnu.tar.gz
7c7d8ad3fc2e28071429a7e25d8235cefddc489ec7d8992c54f4db517f09f5a1  guix-build-132cae44f2d0/output/x86_64-apple-darwin18/SHA256SUMS.part
69767250b6f817cdc424498a60dec00db412c08950eb06d3687814d0442cda02  guix-build-132cae44f2d0/output/x86_64-apple-darwin18/bitcoin-132cae44f2d0-osx-unsigned.dmg
ea42729a477276ac26982a7a22eee940aff567e367fe151c355b21c3a1b5d714  guix-build-132cae44f2d0/output/x86_64-apple-darwin18/bitcoin-132cae44f2d0-osx-unsigned.tar.gz
cb6c48623e78065551cf3ff4ce45dc1f48dab6538dc571d055c9e2b58ceb5178  guix-build-132cae44f2d0/output/x86_64-apple-darwin18/bitcoin-132cae44f2d0-osx64.tar.gz
f49ebd853f247dbbb01ac325f160d37684540e9c6dad9d1be53cc1d1f25b4e13  guix-build-132cae44f2d0/output/x86_64-linux-gnu/SHA256SUMS.part
a4b55db9e3f5f36a7f856da7ad6aba63dcc70f4b3a1a67b5c45681f5b9e35d90  guix-build-132cae44f2d0/output/x86_64-linux-gnu/bitcoin-132cae44f2d0-x86_64-linux-gnu-debug.tar.gz
ba6f9ef3e4ce462019a38df151eed104ab9ebe72eb93fb99e7a54a8c3999a7eb  guix-build-132cae44f2d0/output/x86_64-linux-gnu/bitcoin-132cae44f2d0-x86_64-linux-gnu.tar.gz
4dc9d667ceec5cca1c6fddfa1296bd174da1be04cef015eebe067dc302600ca0  guix-build-132cae44f2d0/output/x86_64-w64-mingw32/SHA256SUMS.part
4d0799e48974fd1fd5023f98e488c66b4a90f753b2de6fd7b214c7193256a10d  guix-build-132cae44f2d0/output/x86_64-w64-mingw32/bitcoin-132cae44f2d0-win-unsigned.tar.gz
a98ca150dd426a66a568cd07038e8c360bbe6b78eecf315bab2b146f42da526c  guix-build-132cae44f2d0/output/x86_64-w64-mingw32/bitcoin-132cae44f2d0-win64-debug.zip
a7647e742faebfa344df195775b20478a0e42ee19cb671508acc9d96394ec2b9  guix-build-132cae44f2d0/output/x86_64-w64-mingw32/bitcoin-132cae44f2d0-win64-setup-unsigned.exe
f202165ef8de36ec8a8c503053e6c6e81f1e4c1e1f8cad3abbb68f045a73fb1d  guix-build-132cae44f2d0/output/x86_64-w64-mingw32/bitcoin-132cae44f2d0-win64.zip

Copy link
Member

@fanquake fanquake left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 132cae4

@fanquake fanquake merged commit d316934 into bitcoin:master Aug 20, 2021
hebasto pushed a commit to hebasto/bitcoin that referenced this pull request Aug 20, 2021
The SHA256SUMS file can be used in a sha256sum -c command to verify
downloaded binaries. However users are likely to download just a single
file and not place this file in the correct directory relative to the
SHA256SUMS file for the simple verification command to work. By not
including the directory name in the SHA256SUMS file, it will be easier
for users to verify downloaded binaries.

Co-authored-by: Carl Dong <contact@carldong.me>

Github-Pull: bitcoin#22654
Rebased-From: fb17c99
hebasto pushed a commit to hebasto/bitcoin that referenced this pull request Aug 20, 2021
The uploaded binaries need to match the same flat directory structure of
the SHA256SUMS file in order for torrent downloaders to be able to
verify the download without moving files. Mention this in the release
process doc.

Github-Pull: bitcoin#22654
Rebased-From: 132cae4
@hebasto hebasto mentioned this pull request Aug 20, 2021
@hebasto
Copy link
Member

hebasto commented Aug 20, 2021

Backported in #22629.

hebasto pushed a commit to hebasto/bitcoin that referenced this pull request Aug 20, 2021
The SHA256SUMS file can be used in a sha256sum -c command to verify
downloaded binaries. However users are likely to download just a single
file and not place this file in the correct directory relative to the
SHA256SUMS file for the simple verification command to work. By not
including the directory name in the SHA256SUMS file, it will be easier
for users to verify downloaded binaries.

Co-authored-by: Carl Dong <contact@carldong.me>

Github-Pull: bitcoin#22654
Rebased-From: fb17c99
hebasto pushed a commit to hebasto/bitcoin that referenced this pull request Aug 20, 2021
The uploaded binaries need to match the same flat directory structure of
the SHA256SUMS file in order for torrent downloaders to be able to
verify the download without moving files. Mention this in the release
process doc.

Github-Pull: bitcoin#22654
Rebased-From: 132cae4
hebasto pushed a commit to hebasto/bitcoin that referenced this pull request Aug 20, 2021
The SHA256SUMS file can be used in a sha256sum -c command to verify
downloaded binaries. However users are likely to download just a single
file and not place this file in the correct directory relative to the
SHA256SUMS file for the simple verification command to work. By not
including the directory name in the SHA256SUMS file, it will be easier
for users to verify downloaded binaries.

Co-authored-by: Carl Dong <contact@carldong.me>

Github-Pull: bitcoin#22654
Rebased-From: fb17c99
hebasto pushed a commit to hebasto/bitcoin that referenced this pull request Aug 20, 2021
The uploaded binaries need to match the same flat directory structure of
the SHA256SUMS file in order for torrent downloaders to be able to
verify the download without moving files. Mention this in the release
process doc.

Github-Pull: bitcoin#22654
Rebased-From: 132cae4
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Aug 20, 2021
laanwj added a commit that referenced this pull request Aug 26, 2021
32e1424 Fix build with Boost 1.77.0 (Rafael Sadowski)
cb34a0a qt: Handle new added plurals in bitcoin_en.ts (Hennadii Stepanov)
068985c doc: Mention the flat directory structure for uploads (Andrew Chow)
27d43e5 guix: Don't include directory name in SHA256SUMS (Andrew Chow)
88fb7e3 test: fix bug in 22686 (S3RK)
63fec7e clientversion: No suffix #if CLIENT_VERSION_IS_RELEASE (Carl Dong)
dfaffbe test: Test for ApproximateBestSubset edge case with too little fees (Andrew Chow)
e86b023 wallet: Assert that enough was selected to cover the fees (Andrew Chow)
ffc81e2 wallet: Use GetSelectionAmount for target value calculations (Andrew Chow)
ce77b45 release: Release with separate SHA256SUMS and sig files (Carl Dong)
cb491bd guix-verify: Non-zero exit code when anything fails (Carl Dong)
6a611d2 gui: ensure external signer option remains disabled without signers (Andrew Chow)
e9b4487 qt: Fix regression in "Encrypt Wallet" menu item (Hennadii Stepanov)
57fce06 consensus/params: simplify ValidDeployment check to avoid gcc warning (Anthony Towns)
e9d30fb ci: Run fuzzer task for the master branch only (Hennadii Stepanov)

Pull request description:

  Backported:

  1) #22730
  1) bitcoin-core/gui#393
  1) #22597
  1) bitcoin-core/gui#396
  1) #22643
  1) #22642
  1) #22685
  1) #22686
  1) #22654
  1) #22742
  1) bitcoin-core/gui#406
  1) #22713

ACKs for top commit:
  laanwj:
    Code list-of-backported-PRs review ACK 32e1424

Tree-SHA512: f5e2dd1be6cdcd39368eeb5d297b3ff4418d0bf2e70c90e59ab4ba1dbf16f773045d877b4997511de58c3aca75a978dcf043e338bad23951557e2a27ccc845f6
fujicoin pushed a commit to fujicoin/fujicoin-22.0 that referenced this pull request Aug 27, 2021
The SHA256SUMS file can be used in a sha256sum -c command to verify
downloaded binaries. However users are likely to download just a single
file and not place this file in the correct directory relative to the
SHA256SUMS file for the simple verification command to work. By not
including the directory name in the SHA256SUMS file, it will be easier
for users to verify downloaded binaries.

Co-authored-by: Carl Dong <contact@carldong.me>

Github-Pull: bitcoin/bitcoin#22654
Rebased-From: fb17c99e35e72f3b21ec3b5473e84c21dc964776
fujicoin added a commit to fujicoin/fujicoin-22.0 that referenced this pull request Aug 27, 2021
The uploaded binaries need to match the same flat directory structure of
the SHA256SUMS file in order for torrent downloaders to be able to
verify the download without moving files. Mention this in the release
process doc.

Github-Pull: bitcoin/bitcoin#22654
Rebased-From: 132cae44f2d031bdaa1e459b92ec89ad585dfc9f
gwillen pushed a commit to gwillen/elements that referenced this pull request Jul 27, 2022
The SHA256SUMS file can be used in a sha256sum -c command to verify
downloaded binaries. However users are likely to download just a single
file and not place this file in the correct directory relative to the
SHA256SUMS file for the simple verification command to work. By not
including the directory name in the SHA256SUMS file, it will be easier
for users to verify downloaded binaries.

Co-authored-by: Carl Dong <contact@carldong.me>

Github-Pull: bitcoin/bitcoin#22654
Rebased-From: fb17c99
gwillen pushed a commit to gwillen/elements that referenced this pull request Jul 27, 2022
The uploaded binaries need to match the same flat directory structure of
the SHA256SUMS file in order for torrent downloaders to be able to
verify the download without moving files. Mention this in the release
process doc.

Github-Pull: bitcoin/bitcoin#22654
Rebased-From: 132cae4
gwillen pushed a commit to gwillen/elements that referenced this pull request Aug 1, 2022
The SHA256SUMS file can be used in a sha256sum -c command to verify
downloaded binaries. However users are likely to download just a single
file and not place this file in the correct directory relative to the
SHA256SUMS file for the simple verification command to work. By not
including the directory name in the SHA256SUMS file, it will be easier
for users to verify downloaded binaries.

Co-authored-by: Carl Dong <contact@carldong.me>

Github-Pull: bitcoin/bitcoin#22654
Rebased-From: fb17c99
gwillen pushed a commit to gwillen/elements that referenced this pull request Aug 1, 2022
The uploaded binaries need to match the same flat directory structure of
the SHA256SUMS file in order for torrent downloaders to be able to
verify the download without moving files. Mention this in the release
process doc.

Github-Pull: bitcoin/bitcoin#22654
Rebased-From: 132cae4
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Aug 20, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants