-
Notifications
You must be signed in to change notification settings - Fork 37.7k
guix: Don't include directory name in SHA256SUMS #22654
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
c1f2435
to
2dabd97
Compare
Why don't you mention this PR in #22634 (comment) ? I have now rewritten my download&verfiy bash script already. It's true that I e.g. do not download into a subfolder "x86_64-linux-gnu", so as you said, I experienced that sha256sum -c --ignore-missing does not work. My workaround was just to do I am not sure if this is good or bad, on one hand I like it that the SHA256SUMS file mirrors the download directory structure, that is clean and I think it would not be feasible to have an SHA256SUMS file in every arch sub dir, because that would require also a signature file for each of them. On the other hand, I agree, that normally one does not download in an arch sub dir. But the latter could be done easily I think, though. |
Concept ACK. Arch is included in the output filenames, therefore, no name clash could be expected. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK 2dabd97, I have reviewed the code and it looks OK, I agree it can be merged.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK 2dabd97
LGTM, although, I would suggest adding the comment proposed here #22654 (comment) to make future code review easier.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re-ACK 568ef8d
Nice comment addition.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please see my comment
Sorry that this is now in the "Conversation" section, I only tried out if one could some kind of "reopen" the conversation in the "Files changed" tab, which was marked as resolved by Achow, instead of only adding a comment to it. Now I cannot find out how to remove this.
Just asking, why arch subfolders at all, if every 22.0rc2 subfolder contains just one file with a unique name which contains the arch, except the folder x86_64-w64-mingw32/ which has two files in it, also without clash? |
As I understand this makes the uploaded binaries have all files in one directory like previous releases? (e.g. https://bitcoincore.org/bin/bitcoin-core-0.21.1/ instead of https://bitcoincore.org/bin/bitcoin-core-22.0/test.rc2/) I didn't think nesting the directory per platform makes anything easier or better, but does add an inconvenient extra click when downloading. So concept ACK.
I would suggest holding off doing any large tool rewrites based on process changes until 22.0 final. Things are still being hammered out. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code change ACK, Concept 0
I am not sure what you are referring to with "this"? |
Having a single-level SHA256SUMS means that we can put the files in one directory again during upload (anyhow, that users are able to verify that without having to move files around or pre-process SHA256SUMS). |
Unless the binary server validates the SHA256SUMS file, I don't think so. I don't think it really matters where the binaries are located. Regardless of the directory structure, users are still going to just download one binary file and the SHA256SUMS file into some Downloads directory and verify them when they are at the same level. |
From IRC:
So I guess the directory structure will need to change for the upload. I will add something to |
c602555
to
56c25f5
Compare
contrib/guix/libexec/build.sh
Outdated
} | xargs realpath --relative-base="$PWD" \ | ||
| xargs sha256sum \ | ||
# Get all of the generated output files and hash them without directory names. | ||
# Because "find" prints filenames with a preceding "./", we need to pass the filenames into "basename" first. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally I think it would be good to leave the preceding ./
path, because it is precise.
The verifying works with it on Linux, so the removing is not generally needed.
I guess it shall be removed to be compatible with OSs that have other directory naming conventions, e.g. like Microsoft products?
I would find a comment helpful, for what reason the ./
is needed to be removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typically SHA256SUMS do not contain paths to the files. Seeing the ./
may be confusing to users who are not familiar with paths.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW here (leading to the doc of md5sum --check
invocation) is the doc of SHA256SUMS file:
The input to this mode of md5sum is usually the output of a prior, checksum-generating run of ‘md5sum’.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, but many people do a visual comparison.
Here's an alternative to 752235f, which does the "basenamification" at attest-time (which I think makes a bit more sense): diff --git a/contrib/guix/guix-attest b/contrib/guix/guix-attest
index 1503c330b2..2e2ac0b411 100755
--- a/contrib/guix/guix-attest
+++ b/contrib/guix/guix-attest
@@ -159,6 +159,18 @@ Hint: You may wish to remove the existing attestations and their signatures by
EOF
}
+# Give a SHA256SUMS file as stdin that has lines like:
+# 0ba536819b221a91d3d42e978be016aac918f40984754d74058aa0c921cd3ea6 a/b/d/c/d/s/bitcoin-22.0rc2-riscv64-linux-gnu.tar.gz
+# ...
+#
+# Replace each line's file name with its basename:
+# 0ba536819b221a91d3d42e978be016aac918f40984754d74058aa0c921cd3ea6 bitcoin-22.0rc2-riscv64-linux-gnu.tar.gz
+# ...
+#
+basenameify_SHA256SUMS() {
+ sed -E 's@(^[[:xdigit:]]{64}[[:space:]]+).+/([^/]+$)@\1\2@'
+}
+
echo "Attesting to build outputs for version: '${VERSION}'"
echo ""
@@ -174,6 +186,7 @@ mkdir -p "$outsigdir"
cat "${noncodesigned_fragments[@]}" \
| sort -u \
| sort -k2 \
+ | basenameify_SHA256SUMS \
> "$temp_noncodesigned"
if [ -e noncodesigned.SHA256SUMS ]; then
# The SHA256SUMS already exists, make sure it's exactly what we
@@ -201,6 +214,7 @@ mkdir -p "$outsigdir"
cat "${sha256sum_fragments[@]}" \
| sort -u \
| sort -k2 \
+ | basenameify_SHA256SUMS \
> "$temp_all"
if [ -e all.SHA256SUMS ]; then
# The SHA256SUMS already exists, make sure it's exactly what we |
Yes I think so. |
The SHA256SUMS file can be used in a sha256sum -c command to verify downloaded binaries. However users are likely to download just a single file and not place this file in the correct directory relative to the SHA256SUMS file for the simple verification command to work. By not including the directory name in the SHA256SUMS file, it will be easier for users to verify downloaded binaries. Co-authored-by: Carl Dong <contact@carldong.me>
Changed to use @dongcarl's suggestion |
56c25f5
to
1822d40
Compare
The uploaded binaries need to match the same flat directory structure of the SHA256SUMS file in order for torrent downloaders to be able to verify the download without moving files. Mention this in the release process doc.
1822d40
to
132cae4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re-ACK 132cae4
Guix builds: 777a1ad25da672e72356d34638fef917649462307e6a857e0c2f4d78b8d3e363 guix-build-132cae44f2d0/output/aarch64-linux-gnu/SHA256SUMS.part
5f307edda9d8785a7051bdc89e667500ddd932324f6a0b225d605a921236cdf5 guix-build-132cae44f2d0/output/aarch64-linux-gnu/bitcoin-132cae44f2d0-aarch64-linux-gnu-debug.tar.gz
99c0fe0aa4952af02384ebd4b84c490351ad462290e4673af5c16d13929d929a guix-build-132cae44f2d0/output/aarch64-linux-gnu/bitcoin-132cae44f2d0-aarch64-linux-gnu.tar.gz
d30698cf2358edd729c9cae72270df510d2d811debbac1fd2619ad5aeed8eab3 guix-build-132cae44f2d0/output/arm-linux-gnueabihf/SHA256SUMS.part
2a39d1e11e63a1662d644e16d8002ebc778acaac04ac23572666fb57f15f1d07 guix-build-132cae44f2d0/output/arm-linux-gnueabihf/bitcoin-132cae44f2d0-arm-linux-gnueabihf-debug.tar.gz
807494da57ef85914988500f3533257c5ae09b9351ad742ab317cb4326a72fed guix-build-132cae44f2d0/output/arm-linux-gnueabihf/bitcoin-132cae44f2d0-arm-linux-gnueabihf.tar.gz
31067da3649769233be6ccf8e0aebf0e09af2d70ae649c42011557b111fd0811 guix-build-132cae44f2d0/output/dist-archive/bitcoin-132cae44f2d0.tar.gz
a9b3aad4050e69ff5bc792c2057934da227e793a7ccd73514031bd8c4b48623c guix-build-132cae44f2d0/output/powerpc64-linux-gnu/SHA256SUMS.part
23a0874ab176e0574351ff60e8a193196b48eb640b0a6f77188d7ceab5f70a3c guix-build-132cae44f2d0/output/powerpc64-linux-gnu/bitcoin-132cae44f2d0-powerpc64-linux-gnu-debug.tar.gz
bd0d1ac1ffe179ba1d60259cdf718b24f72490c95656842bd2df0fc8f44e794d guix-build-132cae44f2d0/output/powerpc64-linux-gnu/bitcoin-132cae44f2d0-powerpc64-linux-gnu.tar.gz
ab4d8038e2c1372d2beaaa5c647ba3b30cfe2e0afb38c2174f1264955dacae25 guix-build-132cae44f2d0/output/powerpc64le-linux-gnu/SHA256SUMS.part
41b493490edf41371fe39d6aaef392bec4f1531e26c746a989729cc1bf54f1a4 guix-build-132cae44f2d0/output/powerpc64le-linux-gnu/bitcoin-132cae44f2d0-powerpc64le-linux-gnu-debug.tar.gz
e6b79baa7ed8410f3d5cb4b2d21278c06ce9c665fd7ffe19c1eb3d2e1c147d91 guix-build-132cae44f2d0/output/powerpc64le-linux-gnu/bitcoin-132cae44f2d0-powerpc64le-linux-gnu.tar.gz
7e5ab13ac196511d14a5cb88272ce377257f0b117e5b69a9d8f64aaab8aea580 guix-build-132cae44f2d0/output/riscv64-linux-gnu/SHA256SUMS.part
155146bca7dbe8b499e8ff22395ea71fd770853e075e63293693e6718b1d15b2 guix-build-132cae44f2d0/output/riscv64-linux-gnu/bitcoin-132cae44f2d0-riscv64-linux-gnu-debug.tar.gz
de943b63789da014f642417800197988350be13bdaec2e6ac4dcd8dea0e26b85 guix-build-132cae44f2d0/output/riscv64-linux-gnu/bitcoin-132cae44f2d0-riscv64-linux-gnu.tar.gz
7c7d8ad3fc2e28071429a7e25d8235cefddc489ec7d8992c54f4db517f09f5a1 guix-build-132cae44f2d0/output/x86_64-apple-darwin18/SHA256SUMS.part
69767250b6f817cdc424498a60dec00db412c08950eb06d3687814d0442cda02 guix-build-132cae44f2d0/output/x86_64-apple-darwin18/bitcoin-132cae44f2d0-osx-unsigned.dmg
ea42729a477276ac26982a7a22eee940aff567e367fe151c355b21c3a1b5d714 guix-build-132cae44f2d0/output/x86_64-apple-darwin18/bitcoin-132cae44f2d0-osx-unsigned.tar.gz
cb6c48623e78065551cf3ff4ce45dc1f48dab6538dc571d055c9e2b58ceb5178 guix-build-132cae44f2d0/output/x86_64-apple-darwin18/bitcoin-132cae44f2d0-osx64.tar.gz
f49ebd853f247dbbb01ac325f160d37684540e9c6dad9d1be53cc1d1f25b4e13 guix-build-132cae44f2d0/output/x86_64-linux-gnu/SHA256SUMS.part
a4b55db9e3f5f36a7f856da7ad6aba63dcc70f4b3a1a67b5c45681f5b9e35d90 guix-build-132cae44f2d0/output/x86_64-linux-gnu/bitcoin-132cae44f2d0-x86_64-linux-gnu-debug.tar.gz
ba6f9ef3e4ce462019a38df151eed104ab9ebe72eb93fb99e7a54a8c3999a7eb guix-build-132cae44f2d0/output/x86_64-linux-gnu/bitcoin-132cae44f2d0-x86_64-linux-gnu.tar.gz
4dc9d667ceec5cca1c6fddfa1296bd174da1be04cef015eebe067dc302600ca0 guix-build-132cae44f2d0/output/x86_64-w64-mingw32/SHA256SUMS.part
4d0799e48974fd1fd5023f98e488c66b4a90f753b2de6fd7b214c7193256a10d guix-build-132cae44f2d0/output/x86_64-w64-mingw32/bitcoin-132cae44f2d0-win-unsigned.tar.gz
a98ca150dd426a66a568cd07038e8c360bbe6b78eecf315bab2b146f42da526c guix-build-132cae44f2d0/output/x86_64-w64-mingw32/bitcoin-132cae44f2d0-win64-debug.zip
a7647e742faebfa344df195775b20478a0e42ee19cb671508acc9d96394ec2b9 guix-build-132cae44f2d0/output/x86_64-w64-mingw32/bitcoin-132cae44f2d0-win64-setup-unsigned.exe
f202165ef8de36ec8a8c503053e6c6e81f1e4c1e1f8cad3abbb68f045a73fb1d guix-build-132cae44f2d0/output/x86_64-w64-mingw32/bitcoin-132cae44f2d0-win64.zip |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK 132cae4
The SHA256SUMS file can be used in a sha256sum -c command to verify downloaded binaries. However users are likely to download just a single file and not place this file in the correct directory relative to the SHA256SUMS file for the simple verification command to work. By not including the directory name in the SHA256SUMS file, it will be easier for users to verify downloaded binaries. Co-authored-by: Carl Dong <contact@carldong.me> Github-Pull: bitcoin#22654 Rebased-From: fb17c99
The uploaded binaries need to match the same flat directory structure of the SHA256SUMS file in order for torrent downloaders to be able to verify the download without moving files. Mention this in the release process doc. Github-Pull: bitcoin#22654 Rebased-From: 132cae4
Backported in #22629. |
The SHA256SUMS file can be used in a sha256sum -c command to verify downloaded binaries. However users are likely to download just a single file and not place this file in the correct directory relative to the SHA256SUMS file for the simple verification command to work. By not including the directory name in the SHA256SUMS file, it will be easier for users to verify downloaded binaries. Co-authored-by: Carl Dong <contact@carldong.me> Github-Pull: bitcoin#22654 Rebased-From: fb17c99
The uploaded binaries need to match the same flat directory structure of the SHA256SUMS file in order for torrent downloaders to be able to verify the download without moving files. Mention this in the release process doc. Github-Pull: bitcoin#22654 Rebased-From: 132cae4
The SHA256SUMS file can be used in a sha256sum -c command to verify downloaded binaries. However users are likely to download just a single file and not place this file in the correct directory relative to the SHA256SUMS file for the simple verification command to work. By not including the directory name in the SHA256SUMS file, it will be easier for users to verify downloaded binaries. Co-authored-by: Carl Dong <contact@carldong.me> Github-Pull: bitcoin#22654 Rebased-From: fb17c99
The uploaded binaries need to match the same flat directory structure of the SHA256SUMS file in order for torrent downloaders to be able to verify the download without moving files. Mention this in the release process doc. Github-Pull: bitcoin#22654 Rebased-From: 132cae4
32e1424 Fix build with Boost 1.77.0 (Rafael Sadowski) cb34a0a qt: Handle new added plurals in bitcoin_en.ts (Hennadii Stepanov) 068985c doc: Mention the flat directory structure for uploads (Andrew Chow) 27d43e5 guix: Don't include directory name in SHA256SUMS (Andrew Chow) 88fb7e3 test: fix bug in 22686 (S3RK) 63fec7e clientversion: No suffix #if CLIENT_VERSION_IS_RELEASE (Carl Dong) dfaffbe test: Test for ApproximateBestSubset edge case with too little fees (Andrew Chow) e86b023 wallet: Assert that enough was selected to cover the fees (Andrew Chow) ffc81e2 wallet: Use GetSelectionAmount for target value calculations (Andrew Chow) ce77b45 release: Release with separate SHA256SUMS and sig files (Carl Dong) cb491bd guix-verify: Non-zero exit code when anything fails (Carl Dong) 6a611d2 gui: ensure external signer option remains disabled without signers (Andrew Chow) e9b4487 qt: Fix regression in "Encrypt Wallet" menu item (Hennadii Stepanov) 57fce06 consensus/params: simplify ValidDeployment check to avoid gcc warning (Anthony Towns) e9d30fb ci: Run fuzzer task for the master branch only (Hennadii Stepanov) Pull request description: Backported: 1) #22730 1) bitcoin-core/gui#393 1) #22597 1) bitcoin-core/gui#396 1) #22643 1) #22642 1) #22685 1) #22686 1) #22654 1) #22742 1) bitcoin-core/gui#406 1) #22713 ACKs for top commit: laanwj: Code list-of-backported-PRs review ACK 32e1424 Tree-SHA512: f5e2dd1be6cdcd39368eeb5d297b3ff4418d0bf2e70c90e59ab4ba1dbf16f773045d877b4997511de58c3aca75a978dcf043e338bad23951557e2a27ccc845f6
The SHA256SUMS file can be used in a sha256sum -c command to verify downloaded binaries. However users are likely to download just a single file and not place this file in the correct directory relative to the SHA256SUMS file for the simple verification command to work. By not including the directory name in the SHA256SUMS file, it will be easier for users to verify downloaded binaries. Co-authored-by: Carl Dong <contact@carldong.me> Github-Pull: bitcoin/bitcoin#22654 Rebased-From: fb17c99e35e72f3b21ec3b5473e84c21dc964776
The uploaded binaries need to match the same flat directory structure of the SHA256SUMS file in order for torrent downloaders to be able to verify the download without moving files. Mention this in the release process doc. Github-Pull: bitcoin/bitcoin#22654 Rebased-From: 132cae44f2d031bdaa1e459b92ec89ad585dfc9f
The SHA256SUMS file can be used in a sha256sum -c command to verify downloaded binaries. However users are likely to download just a single file and not place this file in the correct directory relative to the SHA256SUMS file for the simple verification command to work. By not including the directory name in the SHA256SUMS file, it will be easier for users to verify downloaded binaries. Co-authored-by: Carl Dong <contact@carldong.me> Github-Pull: bitcoin/bitcoin#22654 Rebased-From: fb17c99
The uploaded binaries need to match the same flat directory structure of the SHA256SUMS file in order for torrent downloaders to be able to verify the download without moving files. Mention this in the release process doc. Github-Pull: bitcoin/bitcoin#22654 Rebased-From: 132cae4
The SHA256SUMS file can be used in a sha256sum -c command to verify downloaded binaries. However users are likely to download just a single file and not place this file in the correct directory relative to the SHA256SUMS file for the simple verification command to work. By not including the directory name in the SHA256SUMS file, it will be easier for users to verify downloaded binaries. Co-authored-by: Carl Dong <contact@carldong.me> Github-Pull: bitcoin/bitcoin#22654 Rebased-From: fb17c99
The uploaded binaries need to match the same flat directory structure of the SHA256SUMS file in order for torrent downloaders to be able to verify the download without moving files. Mention this in the release process doc. Github-Pull: bitcoin/bitcoin#22654 Rebased-From: 132cae4
The SHA256SUMS file can be used in a sha256sum -c command to verify downloaded binaries. However users are likely to download just a single file and not place this file in the correct directory relative to the SHA256SUMS file for the simple verification command to work. By not including the directory name in the SHA256SUMS file, it will be easier for users to verify downloaded binaries.