build: Link time garbage collection #18605

fanquake · 2020-04-12T12:28:46Z

Adds the linker flags required to turn on link time garbage collection
to the Linux gitian descriptors and Guix build. After some experimentation
I'm not convinced this is something that we should do for Windows builds
at this stage.

Size comparison for a gitian build of master (df303ce) vs this PR (8c31ef9):

binary	master (`df303ce`)	PR (`8c31ef9`)	saving
bitcoind	10525336	9775672	7%
bitcoin-cli	2055800	1113624	45%
bitcoin-qt	31896456	31114024	2.5%
bitcoin-tx	`2376152`	1372536	42%
bitcoin-wallet	5899168	3199808	45%
test_bitcoin	17063328	15354784	10%

DrahtBot · 2020-04-12T13:07:03Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

laanwj · 2020-04-13T11:14:16Z

Concept ACK.

fanquake · 2020-04-16T00:27:20Z

Rebased for #17929 and added a commit for Guix.

jonasschnelli · 2020-04-16T06:42:00Z

Nice. Concept ACK.
Is there a clang equivalent for --gc-sections?
fdata-sections AFAIK is also supported in clang.
Could the clang linker flag be -dead_strip?

laanwj · 2020-04-16T16:24:35Z

contrib/gitian-descriptors/gitian-linux.yml

-  HOST_CFLAGS="-O2 -g"
-  HOST_CXXFLAGS="-O2 -g"
-  HOST_LDFLAGS_BASE="-static-libstdc++ -Wl,-O2"
+  HOST_CFLAGS="-O2 -g -ffunction-sections -fdata-sections"


These flags are effective for the depends build too, I hope?

fanquake · 2020-04-22T09:36:21Z

I've been doing some experiments with Windows builds, and passing --gc-sections there currently results in an increase in binary size. I still need to test using a more recent version of binutils.

Is there a clang equivalent for --gc-sections?

gc-sections is a linker flag, so as long as Clang forwards it through, and whichever linker you are using understands it, it should work. Note that ld64 on macOS does not.

fdata-sections AFAIK is also supported in clang.

Clang supports -ffunction-sections and -fdata-sections. However only for ELF targets.

-fdata-sections         Place each data in its own section (ELF Only)
-ffunction-sections     Place each function in its own section (ELF Only)

Could the clang linker flag be -dead_strip?

We do currently pass dead_strip to ld64 for our macOS builds; and see size reductions. For example, compiling master with and without:

Binary	dead_strip	no dead_strip
bitcoind	10900836	11741428
bitcoin-qt	28593032	32178676
bitcoin-cli	466896	1065220
bitcoin-tx	1196536	1671900
bitcoin-wallet	3054504	`5581956`

3e38023 scripts: add PE .reloc section check to security-check.py (fanquake) Pull request description: The `ld` in binutils has historically had a few issues with PE binaries, there's a good summary in this [thread](https://sourceware.org/bugzilla/show_bug.cgi?id=19011). One issue in particular was `ld` stripping the `.reloc` section out of PE binaries, even though it's required for functioning ASLR. This was [reported by a Tor developer in 2014](https://sourceware.org/bugzilla/show_bug.cgi?id=17321) and they have been patching their [own binutils](https://gitweb.torproject.org/builders/tor-browser-build.git/tree/projects/binutils) ever since. However their patch only made it into binutils at the [start of this year](https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=dc9bd8c92af67947db44b3cb428c050259b15cd0). It adds an `--enable-reloc-section` flag, which is turned on by default if you are using `--dynamic-base`. In the mean time this issue has also been worked around by other projects, such as FFmpeg, see [this commit](TheRyuu/FFmpeg@91b668a). I have checked our recent supported Windows release binaries, and they do contain a `.reloc` section. From what I understand, we are using all the right compile/linker flags, including `-pie` & `-fPIE`, and have never run into the crashing/entrypoint issues that other projects might have seen. One other thing worth noting here, it how Debian/Ubuntu patch the binutils that they distribute, because that's what we end up using in our gitian builds. In the binutils-mingw-w64 in Bionic (18.04), which we currently use in gitian, PE hardening options/security flags are enabled by default. See the [changelog](https://changelogs.ubuntu.com/changelogs/pool/universe/b/binutils-mingw-w64/binutils-mingw-w64_8ubuntu1/changelog) and the [relevant commit](https://salsa.debian.org/mingw-w64-team/binutils-mingw-w64/-/commit/452b3013b8280cbe35eaeb166a43621b88d5f8b7). However in Focal (20.04), this has now been reversed. PE hardening options are no-longer the default. See the [changelog](https://changelogs.ubuntu.com/changelogs/pool/universe/b/binutils-mingw-w64/binutils-mingw-w64_8.8/changelog) and [relevant commit](https://salsa.debian.org/mingw-w64-team/binutils-mingw-w64/-/commit/7bd8b2fbc242a8c2fc2217f29fd61f94d3babf6f), which cites same .reloc issue mentioned here. Given that we explicitly specify/opt-in to everything that we want to use, the defaults aren't necessarily an issue for us. However I think it highlights the importance of continuing to be explicit about what we want, and not falling-back or relying on upstream. This was also prompted by the possibility of us doing link time garbage collection, see #18579 & #18605. It seemed some sanity checks would be worthwhile in-case the linker goes haywire while garbage collecting. I think Guix is going to bring great benefits when dealing with these kinds of issues. Carl you might have something to say in that regard. ACKs for top commit: dongcarl: ACK 3e38023 Tree-SHA512: af14d63bdb334bde548dd7de3e0946556b7e2598d817b56eb4e75b3f56c705c26aa85dd9783134c4b6a7aeb7cb4de567eed996e94d533d31511f57ed332287da

…-check.py 3e38023 scripts: add PE .reloc section check to security-check.py (fanquake) Pull request description: The `ld` in binutils has historically had a few issues with PE binaries, there's a good summary in this [thread](https://sourceware.org/bugzilla/show_bug.cgi?id=19011). One issue in particular was `ld` stripping the `.reloc` section out of PE binaries, even though it's required for functioning ASLR. This was [reported by a Tor developer in 2014](https://sourceware.org/bugzilla/show_bug.cgi?id=17321) and they have been patching their [own binutils](https://gitweb.torproject.org/builders/tor-browser-build.git/tree/projects/binutils) ever since. However their patch only made it into binutils at the [start of this year](https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=dc9bd8c92af67947db44b3cb428c050259b15cd0). It adds an `--enable-reloc-section` flag, which is turned on by default if you are using `--dynamic-base`. In the mean time this issue has also been worked around by other projects, such as FFmpeg, see [this commit](TheRyuu/FFmpeg@91b668a). I have checked our recent supported Windows release binaries, and they do contain a `.reloc` section. From what I understand, we are using all the right compile/linker flags, including `-pie` & `-fPIE`, and have never run into the crashing/entrypoint issues that other projects might have seen. One other thing worth noting here, it how Debian/Ubuntu patch the binutils that they distribute, because that's what we end up using in our gitian builds. In the binutils-mingw-w64 in Bionic (18.04), which we currently use in gitian, PE hardening options/security flags are enabled by default. See the [changelog](https://changelogs.ubuntu.com/changelogs/pool/universe/b/binutils-mingw-w64/binutils-mingw-w64_8ubuntu1/changelog) and the [relevant commit](https://salsa.debian.org/mingw-w64-team/binutils-mingw-w64/-/commit/452b3013b8280cbe35eaeb166a43621b88d5f8b7). However in Focal (20.04), this has now been reversed. PE hardening options are no-longer the default. See the [changelog](https://changelogs.ubuntu.com/changelogs/pool/universe/b/binutils-mingw-w64/binutils-mingw-w64_8.8/changelog) and [relevant commit](https://salsa.debian.org/mingw-w64-team/binutils-mingw-w64/-/commit/7bd8b2fbc242a8c2fc2217f29fd61f94d3babf6f), which cites same .reloc issue mentioned here. Given that we explicitly specify/opt-in to everything that we want to use, the defaults aren't necessarily an issue for us. However I think it highlights the importance of continuing to be explicit about what we want, and not falling-back or relying on upstream. This was also prompted by the possibility of us doing link time garbage collection, see bitcoin#18579 & bitcoin#18605. It seemed some sanity checks would be worthwhile in-case the linker goes haywire while garbage collecting. I think Guix is going to bring great benefits when dealing with these kinds of issues. Carl you might have something to say in that regard. ACKs for top commit: dongcarl: ACK 3e38023 Tree-SHA512: af14d63bdb334bde548dd7de3e0946556b7e2598d817b56eb4e75b3f56c705c26aa85dd9783134c4b6a7aeb7cb4de567eed996e94d533d31511f57ed332287da

fanquake · 2020-05-22T06:44:35Z

I've updated this to drop the changes for Windows for now. After some more experimenting it wasn't clear that it was working as we'd like and I was still seeing increased binary sizes with newer binutils.

I have also updated the PR description with a comparison of the size differences for a gitian build of master vs this branch.

practicalswift · 2020-09-18T05:25:05Z

Concept ACK

Adds the linker flag required to turn on link time garbage collection to the linux gitian descriptors. Combined with `-ffunction-sections` and `-fdata-sections` this results in a between 5% and 45% reduction in file size for the binaries we are shipping.

Adds the linker flags required to turn on link time garbage collection to the Linux Guix build. Combined with `-ffunction-sections` and `-fdata-sections` this results in a between ~5% and 45% reduction in file size for the binaries we are shipping.

fanquake · 2020-11-22T04:21:56Z

I rebased this and ran some gitian builds to compare master (816132e) and this PR (d05d560) (also included 0.20.1):

# ls -la bitcoin-0.20.1/bin/
2055800 Jul 31 12:01 bitcoin-cli
31843176 Jul 31 12:01 bitcoin-qt
2376152 Jul 31 12:01 bitcoin-tx
5952832 Jul 31 12:01 bitcoin-wallet
10312344 Jul 31 12:01 bitcoind
16702880 Jul 31 12:01 test_bitcoin

# ls -la bitcoin-816132e6eb23/bin/
4689528 Nov 21 06:47 bitcoin-cli
36844872 Nov 21 06:47 bitcoin-qt
4792088 Nov 21 06:47 bitcoin-tx
10000192 Nov 21 06:47 bitcoin-wallet
15277080 Nov 21 06:47 bitcoind
21103040 Nov 21 06:47 test_bitcoin

# ls -la bitcoin-d05d56047a38/bin
4439576 Apr 15  2020 bitcoin-cli
34407752 Apr 15  2020 bitcoin-qt
4505304 Apr 15  2020 bitcoin-tx
7214880 Apr 15  2020 bitcoin-wallet
14953496 Apr 15  2020 bitcoind
20516928 Apr 15  2020 test_bitcoin

All of our shipped binaries have certainly grown a bit since 0.20.1. There's still a few mb to be saved in some cases.

dergoegge · 2020-12-09T13:51:16Z

I confirmed the space savings by running the gitian builds on master and this PR. I also read up on how the used options (functions-sections, data-sections, gc-sections) work and if there any unwanted side effects.

Two questions:

AFAICT the options are not active in the depends build. Why not?

bitcoin/contrib/gitian-descriptors/gitian-linux.yml

Lines 96 to 99 in d05d560

    
             # Build dependencies for each host 
        
             for i in $HOSTS; do 
        
               make ${MAKEOPTS} -C ${BASEPREFIX} HOST="${i}" 
        
             done

The gcc docs warn to only use these options if there is a significant benefit from doing so because certain compiler optimizations wont work while they are enabled. Is it ok to ignore that warning? If so, why?

From https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html:

Only use these options when there are significant benefits from doing so. [...] They prevent optimizations by the compiler and assembler using relative locations inside a translation unit since the locations are unknown until link time. An example of such an optimization is relaxing calls to short call instructions.

68e5aaf build: add `--enable-lto` configuration option (fanquake) Pull request description: It's been 5 years since using LTO was first suggested for use when building Bitcoin Core, and it's time to revisit it again. Compilers, and their LTO implementations, have matured, and Bitcoin Core has come a long way in terms of pruning dependencies which may have proved troublesome (i.e Boost previously had issues when using LTO). We'll have even less Boost code after moving to `std::filesystem` (#20744). Experimenting with LTO came up on IRC last night: > sipa: jamesob: i'm interested in knowing whether "-flto" and/or "-fdata-sections -ffunction-sections -Wl,--gc-sections" are possible/beneficial with our current compiler suite; what would be a good way to have your test infrastructure benchmark things? So this PR just adds the bare minimum to make it easier to configure, compile and perform some bench-marking using `-flto`. This PR doesn't do anything depends wise, however if we decide this is what we want to do, I'll expand the changes here. I had previously had a PR open (#18605) to perform link time garbage collection (`-ffunction-sections -fdata-sections` & `-Wl,--gc-sections`), however moving straight to using LTO would be preferable. Note that our minimum required set of compilers, GCC 8.1 and Clang 7, all support the `-flto` option. Related #18579. Previous discussion: #10616, #14277. Previous related PRs: #10800 (`-flto`), #16791 (ThinLTO). Guix build: ```bash bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum 1f3a7c5be4169aaa444b481d3e65a7bb72da9007fee6e6c416ded2e70f97374b guix-build-68e5aafde3e8/output/aarch64-linux-gnu/SHA256SUMS.part fa8f4cf223d9aaf0b2c1ef55ce61256a19cd1ad7f42b99d0b98c9a52fe6ad8ba guix-build-68e5aafde3e8/output/aarch64-linux-gnu/bitcoin-68e5aafde3e8-aarch64-linux-gnu-debug.tar.gz 9a9967078cd1849b4e85db619e1f55d305c6d44e9e013067c0e8d62c1ba54087 guix-build-68e5aafde3e8/output/aarch64-linux-gnu/bitcoin-68e5aafde3e8-aarch64-linux-gnu.tar.gz 18c71f30722102baaf3dfda67f7c7aac38723510b142e8df8ee7063c5d499368 guix-build-68e5aafde3e8/output/arm-linux-gnueabihf/SHA256SUMS.part 0854cc0d17c045a118df2a24e4cf36d727e7e7e2dea37c2492ee21b71cb79b4b guix-build-68e5aafde3e8/output/arm-linux-gnueabihf/bitcoin-68e5aafde3e8-arm-linux-gnueabihf-debug.tar.gz 215256897dde4e8412ed60473376c694a80c5479fb08039107fb62435f2816ef guix-build-68e5aafde3e8/output/arm-linux-gnueabihf/bitcoin-68e5aafde3e8-arm-linux-gnueabihf.tar.gz 5fad0d9d12bc514ec46ed5d66fd29b7da1376a4a69c3b692936f1ab2356e2f85 guix-build-68e5aafde3e8/output/dist-archive/bitcoin-68e5aafde3e8.tar.gz 4f32989d4ab1946048ca7caee9a983fa875be262282562f5a3e040f4bf92158e guix-build-68e5aafde3e8/output/powerpc64-linux-gnu/SHA256SUMS.part ae45df309ae8ada52891efac0a369a69fed4ab93847a7bc4150a62230df4c8d7 guix-build-68e5aafde3e8/output/powerpc64-linux-gnu/bitcoin-68e5aafde3e8-powerpc64-linux-gnu-debug.tar.gz 0ced227de15cb578567131271e2effe80681b4d7a436c92bf1caec735a576fa4 guix-build-68e5aafde3e8/output/powerpc64-linux-gnu/bitcoin-68e5aafde3e8-powerpc64-linux-gnu.tar.gz 26fc5d2ccc1bc17ee0a146cacada6f4909d90c136ae640c8337332adce414ee0 guix-build-68e5aafde3e8/output/powerpc64le-linux-gnu/SHA256SUMS.part 9956b544d90a62a8ba9fc9dc6b6b7f0efe193357332ec19e88053a89d4aab37e guix-build-68e5aafde3e8/output/powerpc64le-linux-gnu/bitcoin-68e5aafde3e8-powerpc64le-linux-gnu-debug.tar.gz be8e39ceea1d36086ce5fa93bfb138c68d3bdf0dd6950b192dfa27a65cce3836 guix-build-68e5aafde3e8/output/powerpc64le-linux-gnu/bitcoin-68e5aafde3e8-powerpc64le-linux-gnu.tar.gz a7755edc394972885c4c77a7798007e5ba4126b177c4ff6224275c4fb8f3b1c4 guix-build-68e5aafde3e8/output/riscv64-linux-gnu/SHA256SUMS.part b6d252993d8aae7582ad6385fe53c61c54c284c68ece6cb2b2d1ac9554e06139 guix-build-68e5aafde3e8/output/riscv64-linux-gnu/bitcoin-68e5aafde3e8-riscv64-linux-gnu-debug.tar.gz bb4860f3bbd815f800333124ff901d880741792ab47097f49bda3a6931144da0 guix-build-68e5aafde3e8/output/riscv64-linux-gnu/bitcoin-68e5aafde3e8-riscv64-linux-gnu.tar.gz 3dd17deed5c5935fb28b62dfc7afca5caab0d67862cdcbf3337edae73e1d0c4c guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/SHA256SUMS.part fa2d68c54fda0816188c81ce2201a77340b82645da2ffe412526f92c297a82df guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/bitcoin-68e5aafde3e8-osx-unsigned.dmg f6e5accdcd201f522b6426e4d8cc9b3643d4d43a57d268fa0e79ea9a34cfac01 guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/bitcoin-68e5aafde3e8-osx-unsigned.tar.gz 4e5a127df957d1c73b65925d685f6620e7bc5667efcb6dcd98be76effc22fc12 guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/bitcoin-68e5aafde3e8-osx64.tar.gz 56ccd216a69acafacbdc6bae0bdcc1faa50b6a51be1aebfa7068206c88b3241a guix-build-68e5aafde3e8/output/x86_64-linux-gnu/SHA256SUMS.part 77b93dd5fad322636853e5b0244ffafd97cc97f3b4b4ee755d5f830b75d77d13 guix-build-68e5aafde3e8/output/x86_64-linux-gnu/bitcoin-68e5aafde3e8-x86_64-linux-gnu-debug.tar.gz 1feda932fc127b900316a232432b91e46e57ee12a81e12a7d888fdc3296219c1 guix-build-68e5aafde3e8/output/x86_64-linux-gnu/bitcoin-68e5aafde3e8-x86_64-linux-gnu.tar.gz aa7c53ab4164b3736049065c3c24391fc5bd7f26b4bda4aa877c378f0636a125 guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/SHA256SUMS.part 5e76148e67aef7e91e70074bfadc08e94373449ac3b966f4343b04d230c778fd guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win-unsigned.tar.gz 34123e3d818beeb70113caeda66945bc7cb9d9e987515d5b149bd17b4b38da90 guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win64-debug.zip 2bba7f40a2b23c6ea3d47c4f564ab54201bf27f7f57103a98cc9bceea4e70c4d guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win64-setup-unsigned.exe 0e7e124144af4a92a4344cf70a3b7c06fbd2b8782aee7ede7263893afa3a5ef0 guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win64.zip ``` ACKs for top commit: laanwj: Code review ACK 68e5aaf Tree-SHA512: 5c25249cc178b9d54159e268390c974b739df9458d773e23c14b14d808f87f7afe314058b3c068601a9132042321973b0c9b6f81becb925665eca2738ae9a613

68e5aaf build: add `--enable-lto` configuration option (fanquake) Pull request description: It's been 5 years since using LTO was first suggested for use when building Bitcoin Core, and it's time to revisit it again. Compilers, and their LTO implementations, have matured, and Bitcoin Core has come a long way in terms of pruning dependencies which may have proved troublesome (i.e Boost previously had issues when using LTO). We'll have even less Boost code after moving to `std::filesystem` (bitcoin#20744). Experimenting with LTO came up on IRC last night: > sipa: jamesob: i'm interested in knowing whether "-flto" and/or "-fdata-sections -ffunction-sections -Wl,--gc-sections" are possible/beneficial with our current compiler suite; what would be a good way to have your test infrastructure benchmark things? So this PR just adds the bare minimum to make it easier to configure, compile and perform some bench-marking using `-flto`. This PR doesn't do anything depends wise, however if we decide this is what we want to do, I'll expand the changes here. I had previously had a PR open (bitcoin#18605) to perform link time garbage collection (`-ffunction-sections -fdata-sections` & `-Wl,--gc-sections`), however moving straight to using LTO would be preferable. Note that our minimum required set of compilers, GCC 8.1 and Clang 7, all support the `-flto` option. Related bitcoin#18579. Previous discussion: bitcoin#10616, bitcoin#14277. Previous related PRs: bitcoin#10800 (`-flto`), bitcoin#16791 (ThinLTO). Guix build: ```bash bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum 1f3a7c5be4169aaa444b481d3e65a7bb72da9007fee6e6c416ded2e70f97374b guix-build-68e5aafde3e8/output/aarch64-linux-gnu/SHA256SUMS.part fa8f4cf223d9aaf0b2c1ef55ce61256a19cd1ad7f42b99d0b98c9a52fe6ad8ba guix-build-68e5aafde3e8/output/aarch64-linux-gnu/bitcoin-68e5aafde3e8-aarch64-linux-gnu-debug.tar.gz 9a9967078cd1849b4e85db619e1f55d305c6d44e9e013067c0e8d62c1ba54087 guix-build-68e5aafde3e8/output/aarch64-linux-gnu/bitcoin-68e5aafde3e8-aarch64-linux-gnu.tar.gz 18c71f30722102baaf3dfda67f7c7aac38723510b142e8df8ee7063c5d499368 guix-build-68e5aafde3e8/output/arm-linux-gnueabihf/SHA256SUMS.part 0854cc0d17c045a118df2a24e4cf36d727e7e7e2dea37c2492ee21b71cb79b4b guix-build-68e5aafde3e8/output/arm-linux-gnueabihf/bitcoin-68e5aafde3e8-arm-linux-gnueabihf-debug.tar.gz 215256897dde4e8412ed60473376c694a80c5479fb08039107fb62435f2816ef guix-build-68e5aafde3e8/output/arm-linux-gnueabihf/bitcoin-68e5aafde3e8-arm-linux-gnueabihf.tar.gz 5fad0d9d12bc514ec46ed5d66fd29b7da1376a4a69c3b692936f1ab2356e2f85 guix-build-68e5aafde3e8/output/dist-archive/bitcoin-68e5aafde3e8.tar.gz 4f32989d4ab1946048ca7caee9a983fa875be262282562f5a3e040f4bf92158e guix-build-68e5aafde3e8/output/powerpc64-linux-gnu/SHA256SUMS.part ae45df309ae8ada52891efac0a369a69fed4ab93847a7bc4150a62230df4c8d7 guix-build-68e5aafde3e8/output/powerpc64-linux-gnu/bitcoin-68e5aafde3e8-powerpc64-linux-gnu-debug.tar.gz 0ced227de15cb578567131271e2effe80681b4d7a436c92bf1caec735a576fa4 guix-build-68e5aafde3e8/output/powerpc64-linux-gnu/bitcoin-68e5aafde3e8-powerpc64-linux-gnu.tar.gz 26fc5d2ccc1bc17ee0a146cacada6f4909d90c136ae640c8337332adce414ee0 guix-build-68e5aafde3e8/output/powerpc64le-linux-gnu/SHA256SUMS.part 9956b544d90a62a8ba9fc9dc6b6b7f0efe193357332ec19e88053a89d4aab37e guix-build-68e5aafde3e8/output/powerpc64le-linux-gnu/bitcoin-68e5aafde3e8-powerpc64le-linux-gnu-debug.tar.gz be8e39ceea1d36086ce5fa93bfb138c68d3bdf0dd6950b192dfa27a65cce3836 guix-build-68e5aafde3e8/output/powerpc64le-linux-gnu/bitcoin-68e5aafde3e8-powerpc64le-linux-gnu.tar.gz a7755edc394972885c4c77a7798007e5ba4126b177c4ff6224275c4fb8f3b1c4 guix-build-68e5aafde3e8/output/riscv64-linux-gnu/SHA256SUMS.part b6d252993d8aae7582ad6385fe53c61c54c284c68ece6cb2b2d1ac9554e06139 guix-build-68e5aafde3e8/output/riscv64-linux-gnu/bitcoin-68e5aafde3e8-riscv64-linux-gnu-debug.tar.gz bb4860f3bbd815f800333124ff901d880741792ab47097f49bda3a6931144da0 guix-build-68e5aafde3e8/output/riscv64-linux-gnu/bitcoin-68e5aafde3e8-riscv64-linux-gnu.tar.gz 3dd17deed5c5935fb28b62dfc7afca5caab0d67862cdcbf3337edae73e1d0c4c guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/SHA256SUMS.part fa2d68c54fda0816188c81ce2201a77340b82645da2ffe412526f92c297a82df guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/bitcoin-68e5aafde3e8-osx-unsigned.dmg f6e5accdcd201f522b6426e4d8cc9b3643d4d43a57d268fa0e79ea9a34cfac01 guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/bitcoin-68e5aafde3e8-osx-unsigned.tar.gz 4e5a127df957d1c73b65925d685f6620e7bc5667efcb6dcd98be76effc22fc12 guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/bitcoin-68e5aafde3e8-osx64.tar.gz 56ccd216a69acafacbdc6bae0bdcc1faa50b6a51be1aebfa7068206c88b3241a guix-build-68e5aafde3e8/output/x86_64-linux-gnu/SHA256SUMS.part 77b93dd5fad322636853e5b0244ffafd97cc97f3b4b4ee755d5f830b75d77d13 guix-build-68e5aafde3e8/output/x86_64-linux-gnu/bitcoin-68e5aafde3e8-x86_64-linux-gnu-debug.tar.gz 1feda932fc127b900316a232432b91e46e57ee12a81e12a7d888fdc3296219c1 guix-build-68e5aafde3e8/output/x86_64-linux-gnu/bitcoin-68e5aafde3e8-x86_64-linux-gnu.tar.gz aa7c53ab4164b3736049065c3c24391fc5bd7f26b4bda4aa877c378f0636a125 guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/SHA256SUMS.part 5e76148e67aef7e91e70074bfadc08e94373449ac3b966f4343b04d230c778fd guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win-unsigned.tar.gz 34123e3d818beeb70113caeda66945bc7cb9d9e987515d5b149bd17b4b38da90 guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win64-debug.zip 2bba7f40a2b23c6ea3d47c4f564ab54201bf27f7f57103a98cc9bceea4e70c4d guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win64-setup-unsigned.exe 0e7e124144af4a92a4344cf70a3b7c06fbd2b8782aee7ede7263893afa3a5ef0 guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win64.zip ``` ACKs for top commit: laanwj: Code review ACK 68e5aaf Tree-SHA512: 5c25249cc178b9d54159e268390c974b739df9458d773e23c14b14d808f87f7afe314058b3c068601a9132042321973b0c9b6f81becb925665eca2738ae9a613

…-check.py 3e38023 scripts: add PE .reloc section check to security-check.py (fanquake) Pull request description: The `ld` in binutils has historically had a few issues with PE binaries, there's a good summary in this [thread](https://sourceware.org/bugzilla/show_bug.cgi?id=19011). One issue in particular was `ld` stripping the `.reloc` section out of PE binaries, even though it's required for functioning ASLR. This was [reported by a Tor developer in 2014](https://sourceware.org/bugzilla/show_bug.cgi?id=17321) and they have been patching their [own binutils](https://gitweb.torproject.org/builders/tor-browser-build.git/tree/projects/binutils) ever since. However their patch only made it into binutils at the [start of this year](https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=dc9bd8c92af67947db44b3cb428c050259b15cd0). It adds an `--enable-reloc-section` flag, which is turned on by default if you are using `--dynamic-base`. In the mean time this issue has also been worked around by other projects, such as FFmpeg, see [this commit](TheRyuu/FFmpeg@91b668a). I have checked our recent supported Windows release binaries, and they do contain a `.reloc` section. From what I understand, we are using all the right compile/linker flags, including `-pie` & `-fPIE`, and have never run into the crashing/entrypoint issues that other projects might have seen. One other thing worth noting here, it how Debian/Ubuntu patch the binutils that they distribute, because that's what we end up using in our gitian builds. In the binutils-mingw-w64 in Bionic (18.04), which we currently use in gitian, PE hardening options/security flags are enabled by default. See the [changelog](https://changelogs.ubuntu.com/changelogs/pool/universe/b/binutils-mingw-w64/binutils-mingw-w64_8ubuntu1/changelog) and the [relevant commit](https://salsa.debian.org/mingw-w64-team/binutils-mingw-w64/-/commit/452b3013b8280cbe35eaeb166a43621b88d5f8b7). However in Focal (20.04), this has now been reversed. PE hardening options are no-longer the default. See the [changelog](https://changelogs.ubuntu.com/changelogs/pool/universe/b/binutils-mingw-w64/binutils-mingw-w64_8.8/changelog) and [relevant commit](https://salsa.debian.org/mingw-w64-team/binutils-mingw-w64/-/commit/7bd8b2fbc242a8c2fc2217f29fd61f94d3babf6f), which cites same .reloc issue mentioned here. Given that we explicitly specify/opt-in to everything that we want to use, the defaults aren't necessarily an issue for us. However I think it highlights the importance of continuing to be explicit about what we want, and not falling-back or relying on upstream. This was also prompted by the possibility of us doing link time garbage collection, see bitcoin#18579 & bitcoin#18605. It seemed some sanity checks would be worthwhile in-case the linker goes haywire while garbage collecting. I think Guix is going to bring great benefits when dealing with these kinds of issues. Carl you might have something to say in that regard. ACKs for top commit: dongcarl: ACK 3e38023 Tree-SHA512: af14d63bdb334bde548dd7de3e0946556b7e2598d817b56eb4e75b3f56c705c26aa85dd9783134c4b6a7aeb7cb4de567eed996e94d533d31511f57ed332287da

fanquake added Build system Needs gitian build labels Apr 12, 2020

This was referenced Apr 12, 2020

build: Drop make dist in gitian builds #18556

Merged

build: add linker optimisation flags to gitian & guix (Linux) #17929

Merged

gitian-linux: Build binaries for 64-bit POWER #14066

Closed

DrahtBot removed the Needs gitian build label Apr 13, 2020

fanquake mentioned this pull request Apr 14, 2020

scripts: add PE .reloc section check to security-check.py #18629

Merged

fanquake force-pushed the link_time_garbage_collection branch from 73ca7e1 to 9f2e435 Compare April 15, 2020 23:28

laanwj mentioned this pull request Apr 16, 2020

Compress PNG images with zopflipng. #18661

Closed

laanwj reviewed Apr 16, 2020

View reviewed changes

DrahtBot mentioned this pull request Apr 22, 2020

guix: Make source tarball using git-archive #18741

Merged

fanquake changed the title ~~[WIP] build: Link time garbage collection~~ build: Link time garbage collection May 22, 2020

fanquake force-pushed the link_time_garbage_collection branch from 9f2e435 to 0841e11 Compare May 22, 2020 06:32

fanquake added Needs gitian build DrahtBot Guix build requested labels May 22, 2020

fanquake marked this pull request as ready for review May 22, 2020 06:38

fanquake force-pushed the link_time_garbage_collection branch from 0841e11 to caaa77e Compare May 22, 2020 06:40

fanquake force-pushed the link_time_garbage_collection branch from caaa77e to 8c31ef9 Compare May 22, 2020 06:50

DrahtBot removed the Needs gitian build label May 25, 2020

DrahtBot removed the DrahtBot Guix build requested label Jun 18, 2020

bitcoin deleted a comment from DrahtBot Jun 18, 2020

bitcoin deleted a comment from DrahtBot Sep 24, 2020

fanquake added 2 commits November 22, 2020 10:05

fanquake force-pushed the link_time_garbage_collection branch from 8c31ef9 to d05d560 Compare November 22, 2020 03:33

DrahtBot mentioned this pull request Jan 19, 2021

gitian-linux: Build binaries for 64-bit POWER (continued) #20963

Merged

fanquake closed this Feb 17, 2021

fanquake mentioned this pull request Oct 1, 2021

build: add --enable-lto configuration option #23152

Merged

fanquake mentioned this pull request Aug 3, 2022

Build with -ffunction-sections -fdata-sections -Wl,--gc-sections ? #18579

Closed

bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022

fanquake deleted the link_time_garbage_collection branch November 9, 2022 16:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

build: Link time garbage collection #18605

build: Link time garbage collection #18605

Uh oh!

fanquake commented Apr 12, 2020 •

edited

Loading

Uh oh!

DrahtBot commented Apr 12, 2020 •

edited

Loading

Uh oh!

laanwj commented Apr 13, 2020

Uh oh!

fanquake commented Apr 16, 2020

Uh oh!

jonasschnelli commented Apr 16, 2020

Uh oh!

laanwj Apr 16, 2020

Uh oh!

fanquake commented Apr 22, 2020

Uh oh!

fanquake commented May 22, 2020

Uh oh!

practicalswift commented Sep 18, 2020

Uh oh!

fanquake commented Nov 22, 2020

Uh oh!

dergoegge commented Dec 9, 2020

Uh oh!

Uh oh!

build: Link time garbage collection #18605

build: Link time garbage collection #18605

Uh oh!

Conversation

fanquake commented Apr 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Apr 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Conflicts

Uh oh!

laanwj commented Apr 13, 2020

Uh oh!

fanquake commented Apr 16, 2020

Uh oh!

jonasschnelli commented Apr 16, 2020

Uh oh!

laanwj Apr 16, 2020

Choose a reason for hiding this comment

Uh oh!

fanquake commented Apr 22, 2020

Uh oh!

fanquake commented May 22, 2020

Uh oh!

practicalswift commented Sep 18, 2020

Uh oh!

fanquake commented Nov 22, 2020

Uh oh!

dergoegge commented Dec 9, 2020

Uh oh!

Uh oh!

fanquake commented Apr 12, 2020 •

edited

Loading

DrahtBot commented Apr 12, 2020 •

edited

Loading