guix: Test security-check sanity before performing them #20980

dongcarl · 2021-01-21T19:07:26Z

These changes allow us to make use of the test-security-check target to check the sanity
of our security/symbol checking suite before running them.

fanquake

Concept ACK on running the test-security-check target.

Could link to #18629 in 02706aa for some additional context.

Can you also rebase on master now that macOS support is merged.

fanquake · 2021-01-22T04:26:25Z

contrib/devtools/security-check.py

-OBJDUMP_CMD = os.getenv('OBJDUMP', '/usr/bin/objdump')
-OTOOL_CMD = os.getenv('OTOOL', '/usr/bin/otool')
+OBJDUMP_CMD = os.getenv('OBJDUMP', shutil.which('objdump'))
+OTOOL_CMD = os.getenv('OTOOL', shutil.which('otool'))


If you are going to change these, you'll have to fixup the mypy issues (causing the lint job to fail):

contrib/devtools/symbol-check.py:198: error: List item 0 has incompatible type "Optional[str]"; expected "Union[bytes, str, _PathLike[Any]]" contrib/devtools/symbol-check.py:219: error: List item 0 has incompatible type "Optional[str]"; expected "Union[bytes, str, _PathLike[Any]]" Found 2 errors in 1 file (checked 189 source files) ^---- failure generated from test/lint/lint-python.sh

It's unhappy because we've got an Optional[str] (from shutil.which) being added into the first argument of subprocess.Popen.

Not sure how best to fix...

Naive fix:

diff --git a/contrib/devtools/symbol-check.py b/contrib/devtools/symbol-check.py index 52f04e8cdf..e85f5b5fd4 100755 --- a/contrib/devtools/symbol-check.py +++ b/contrib/devtools/symbol-check.py @@ -52,9 +52,9 @@ IGNORE_EXPORTS = { '_edata', '_end', '__end__', '_init', '__bss_start', '__bss_start__', '_bss_end__', '__bss_end__', '_fini', '_IO_stdin_used', 'stdin', 'stdout', 'stderr', 'environ', '_environ', '__environ', } -CPPFILT_CMD = os.getenv('CPPFILT', shutil.which('c++filt')) -OBJDUMP_CMD = os.getenv('OBJDUMP', shutil.which('objdump')) -OTOOL_CMD = os.getenv('OTOOL', shutil.which('otool')) +CPPFILT_CMD = os.getenv('CPPFILT', shutil.which('c++filt')) # type: ignore[list-item] +OBJDUMP_CMD = os.getenv('OBJDUMP', shutil.which('objdump')) # type: ignore[list-item] +OTOOL_CMD = os.getenv('OTOOL', shutil.which('otool')) # type: ignore[list-item] # Allowed NEEDED libraries ELF_ALLOWED_LIBRARIES = {

Makefile.am

contrib/guix/libexec/build.sh

practicalswift · 2021-01-26T11:11:43Z

Concept ACK on sanity checking test before testing

maflcko · 2021-01-30T07:22:32Z

make[1]: Leaving directory '/distsrc-base/distsrc-65f9b3f774df-x86_64-apple-darwin18'
+ make test-security-check V=1
OTOOL=/bitcoin/depends/x86_64-apple-darwin18/native/bin/x86_64-apple-darwin18-otool /gnu/store/skvjjmxwgy7yjn1jyc5w6z6lmjs6rsjb-profile/bin/python3.7 ./contrib/devtools/test-security-check.py TestSecurityChecks.test_MACHO
ld: unrecognized -a option `llow_stack_execute'
clang-8: error: linker command failed with exit code 1 (use -v to see invocation)
E
======================================================================
ERROR: test_MACHO (__main__.TestSecurityChecks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./contrib/devtools/test-security-check.py", line 70, in test_MACHO
    self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fno-stack-protector']),
  File "./contrib/devtools/test-security-check.py", line 23, in call_security_check
    subprocess.run([cc,source,'-o',executable] + options, check=True)
  File "/gnu/store/hhi58l8s977qv3rvsvs7s9njzy2vpjaa-python-3.7.4/lib/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['clang', 'test1.c', '-o', 'test1', '-Wl,-no_pie', '-Wl,-flat_namespace', '-Wl,-allow_stack_execute', '-fno-stack-protector']' returned non-zero exit status 1.

----------------------------------------------------------------------
Ran 1 test in 0.380s

FAILED (errors=1)
make: *** [Makefile:1429: test-security-check] Error 1

2ecaf21 gitian: remove execstack workaround for ricv64 & powerpc64le (fanquake) 5baff2b build: use focal in gitian descriptors (fanquake) Pull request description: This PR changes the gitian descriptors to use Ubuntu Focal (20.04), over Bionic (18.04), moving from GCC 7.5 to GCC 8.4 for native Linux builds, mingw-w64 GCC 7.3 to mingw-w64 GCC 9.3 for Windows builds, while continuing to use GCC 8.4 for all cross builds and Clang 8.0.0 for macOS builds. It also drops the `-Wl,-z,noexecstack` workaround we've been using for the riscv64 and powerpc64le hosts, as it's no-longer needed. One new package is installed in the osx build, `libtinfo5`, as libtinfo5.so is required by our downloaded Clang 8. A bump to Focal will at least be required if we want to update to a newer Qt (5.15, #19716) for 22.0, as we need a newer version of [`g++-mingw-w64`](https://packages.ubuntu.com/focal/g++-mingw-w64-x86-64) and the [`mingw-w64`](https://mingw-w64.org/doku.php) headers. This can still be done while continuing to use GCC 8.4 for Linux builds (see below), however the newer `g++-mingw-w64` will be based off of GCC 9.3. **Some considerations** GCC 9 is affected by #20005 "memcmp with constants that contain zero bytes are broken in GCC", and the newer `g++-mingw-w64` will be based off of GCC 9.3. The `--no-*` variants of the Windows linker flags (i.e `--no-dynamicbase`) we use to [test our `security-check.py` script](https://github.com/bitcoin/bitcoin/blob/16b784d953365bb2d7ae65acd2b20a79ef8ba7b6/contrib/devtools/test-security-check.py#L53) are not patched into the mingw binutils in Focal (they have been re-added in Groovy (20.10)). This isn't currently an issue, however, we might add a call to `test-security-check` for Guix (#20980), and if we wanted to do the same for gitian, it would not work. Note how it's quite "easy" for us to apply the `--no-*` variant patch to our Guix build; it would be quite a bit harder to do in Gitian. Gitian Builds @ 2ecaf21 #### Linux ```bash 8882ea78486fbae4fac574b9089eb1107c6372d0dd7dfcda4f0f930576f9d6c1 bitcoin-2ecaf214331b-aarch64-linux-gnu-debug.tar.gz 50a9e30943b4eee5163edff3331241e745ff32a2c4463c21a6fdc5986e2d0383 bitcoin-2ecaf214331b-aarch64-linux-gnu.tar.gz ec4e55a447fddf033fee33cd5f22bfeda3c3612f059194bcf6238859f7989d7a bitcoin-2ecaf214331b-arm-linux-gnueabihf-debug.tar.gz 444fe1b3b933c00bcbd4a9d86888cff3b61c1215b1debccd2843e842d1224777 bitcoin-2ecaf214331b-arm-linux-gnueabihf.tar.gz 88e486ff465980dc1a4aab9687d142ec6f727ed2c52cf539f69db2877dee83b2 bitcoin-2ecaf214331b-powerpc64-linux-gnu-debug.tar.gz 66144ac264c65cada9d86446e6026c85b04fb88198b8f41b42840f6031db3e6c bitcoin-2ecaf214331b-powerpc64-linux-gnu.tar.gz 34bcc13d78d929d575e34e77a6672f23ca7ea23230b28ec2eed563889352ba86 bitcoin-2ecaf214331b-powerpc64le-linux-gnu-debug.tar.gz b4c5f959664f3063df4330edfe343c17120eb6b556ee1c15c4aeb2c1c54ffd49 bitcoin-2ecaf214331b-powerpc64le-linux-gnu.tar.gz 918fa72ab6f6ebce4e9663c93f72fe26651c260477cbb54749f7eb61438b5cc1 bitcoin-2ecaf214331b-riscv64-linux-gnu-debug.tar.gz f704f9f8c053ffe37d854e2e81e0f4c0614c435dad7f5d82518c681b73a76ae6 bitcoin-2ecaf214331b-riscv64-linux-gnu.tar.gz b59e3a62f1df9d79f30e916b3c9655f654036fe3a420040c53acc8dd9f4162c5 bitcoin-2ecaf214331b-x86_64-linux-gnu-debug.tar.gz a4dc9ca877cc97544e65db11be38406d16f15d74fcdcd2318bb92474729bc60d bitcoin-2ecaf214331b-x86_64-linux-gnu.tar.gz b40ba2d5da498330ade92a4ccebcceb1452b94c8ffeacb336f87e93b5c88d8af src/bitcoin-2ecaf214331b.tar.gz af6ebc91147778e4e6705eade62608dde4d6e60522d79087fa9129bdb7c01199 bitcoin-core-linux-22-res.yml ``` #### Windows ```bash 121a3970a6911cb8c453b2ce37d03f6cbb43333e29db8fa516c68563fb367f43 bitcoin-2ecaf214331b-win-unsigned.tar.gz 6294e9efebe935092f9ba119dc60ad4094f18b51c4181324e54d3057524d6101 bitcoin-2ecaf214331b-win64-debug.zip 5b5a236b63e67f5f6c07ad9aa716aa7b72fb63722c96798b332c6d164738f9cf bitcoin-2ecaf214331b-win64-setup-unsigned.exe c1fa5894c5e02a201637567c80b9bde9024f44673dcd06fd4d489c1709179279 bitcoin-2ecaf214331b-win64.zip b40ba2d5da498330ade92a4ccebcceb1452b94c8ffeacb336f87e93b5c88d8af src/bitcoin-2ecaf214331b.tar.gz 665fd7eb61aed368150db58a254f15fb5efb51a4efa5abcc52571cb7a1a5de22 bitcoin-core-win-22-res.yml ``` #### macOS ```bash 6a1deae7662aa782baa82a42590f862c6bcdc4f4e38daa9b8c2a9eed1fbb5397 bitcoin-2ecaf214331b-osx-unsigned.dmg 1ee843266e84928a4323fa255c833528c2617a2c9fd2f98fb26ba19bbfc1227b bitcoin-2ecaf214331b-osx-unsigned.tar.gz 097b64dadc167d8e5b733421bf1541a40760ad952990f7cf3f35adc6ae2616d0 bitcoin-2ecaf214331b-osx64.tar.gz b40ba2d5da498330ade92a4ccebcceb1452b94c8ffeacb336f87e93b5c88d8af src/bitcoin-2ecaf214331b.tar.gz 6e378fb543928e40c7119b96be6ff773d38506a9a888f8b02c7f1b8a0801a80e bitcoin-core-osx-22-res.yml ``` ACKs for top commit: laanwj: Build script changes review ACK 2ecaf21 Tree-SHA512: 975d5830b787d2e08988f43cbc6e839294171c1d94c8219636308b05f9b77041421612ae67be24a631674670cfc9c2d96d8177f2b3158a78fc3deea19631febf

2ecaf21 gitian: remove execstack workaround for ricv64 & powerpc64le (fanquake) 5baff2b build: use focal in gitian descriptors (fanquake) Pull request description: This PR changes the gitian descriptors to use Ubuntu Focal (20.04), over Bionic (18.04), moving from GCC 7.5 to GCC 8.4 for native Linux builds, mingw-w64 GCC 7.3 to mingw-w64 GCC 9.3 for Windows builds, while continuing to use GCC 8.4 for all cross builds and Clang 8.0.0 for macOS builds. It also drops the `-Wl,-z,noexecstack` workaround we've been using for the riscv64 and powerpc64le hosts, as it's no-longer needed. One new package is installed in the osx build, `libtinfo5`, as libtinfo5.so is required by our downloaded Clang 8. A bump to Focal will at least be required if we want to update to a newer Qt (5.15, bitcoin#19716) for 22.0, as we need a newer version of [`g++-mingw-w64`](https://packages.ubuntu.com/focal/g++-mingw-w64-x86-64) and the [`mingw-w64`](https://mingw-w64.org/doku.php) headers. This can still be done while continuing to use GCC 8.4 for Linux builds (see below), however the newer `g++-mingw-w64` will be based off of GCC 9.3. **Some considerations** GCC 9 is affected by bitcoin#20005 "memcmp with constants that contain zero bytes are broken in GCC", and the newer `g++-mingw-w64` will be based off of GCC 9.3. The `--no-*` variants of the Windows linker flags (i.e `--no-dynamicbase`) we use to [test our `security-check.py` script](https://github.com/bitcoin/bitcoin/blob/16b784d953365bb2d7ae65acd2b20a79ef8ba7b6/contrib/devtools/test-security-check.py#L53) are not patched into the mingw binutils in Focal (they have been re-added in Groovy (20.10)). This isn't currently an issue, however, we might add a call to `test-security-check` for Guix (bitcoin#20980), and if we wanted to do the same for gitian, it would not work. Note how it's quite "easy" for us to apply the `--no-*` variant patch to our Guix build; it would be quite a bit harder to do in Gitian. Gitian Builds @ 2ecaf21 #### Linux ```bash 8882ea78486fbae4fac574b9089eb1107c6372d0dd7dfcda4f0f930576f9d6c1 bitcoin-2ecaf214331b-aarch64-linux-gnu-debug.tar.gz 50a9e30943b4eee5163edff3331241e745ff32a2c4463c21a6fdc5986e2d0383 bitcoin-2ecaf214331b-aarch64-linux-gnu.tar.gz ec4e55a447fddf033fee33cd5f22bfeda3c3612f059194bcf6238859f7989d7a bitcoin-2ecaf214331b-arm-linux-gnueabihf-debug.tar.gz 444fe1b3b933c00bcbd4a9d86888cff3b61c1215b1debccd2843e842d1224777 bitcoin-2ecaf214331b-arm-linux-gnueabihf.tar.gz 88e486ff465980dc1a4aab9687d142ec6f727ed2c52cf539f69db2877dee83b2 bitcoin-2ecaf214331b-powerpc64-linux-gnu-debug.tar.gz 66144ac264c65cada9d86446e6026c85b04fb88198b8f41b42840f6031db3e6c bitcoin-2ecaf214331b-powerpc64-linux-gnu.tar.gz 34bcc13d78d929d575e34e77a6672f23ca7ea23230b28ec2eed563889352ba86 bitcoin-2ecaf214331b-powerpc64le-linux-gnu-debug.tar.gz b4c5f959664f3063df4330edfe343c17120eb6b556ee1c15c4aeb2c1c54ffd49 bitcoin-2ecaf214331b-powerpc64le-linux-gnu.tar.gz 918fa72ab6f6ebce4e9663c93f72fe26651c260477cbb54749f7eb61438b5cc1 bitcoin-2ecaf214331b-riscv64-linux-gnu-debug.tar.gz f704f9f8c053ffe37d854e2e81e0f4c0614c435dad7f5d82518c681b73a76ae6 bitcoin-2ecaf214331b-riscv64-linux-gnu.tar.gz b59e3a62f1df9d79f30e916b3c9655f654036fe3a420040c53acc8dd9f4162c5 bitcoin-2ecaf214331b-x86_64-linux-gnu-debug.tar.gz a4dc9ca877cc97544e65db11be38406d16f15d74fcdcd2318bb92474729bc60d bitcoin-2ecaf214331b-x86_64-linux-gnu.tar.gz b40ba2d5da498330ade92a4ccebcceb1452b94c8ffeacb336f87e93b5c88d8af src/bitcoin-2ecaf214331b.tar.gz af6ebc91147778e4e6705eade62608dde4d6e60522d79087fa9129bdb7c01199 bitcoin-core-linux-22-res.yml ``` #### Windows ```bash 121a3970a6911cb8c453b2ce37d03f6cbb43333e29db8fa516c68563fb367f43 bitcoin-2ecaf214331b-win-unsigned.tar.gz 6294e9efebe935092f9ba119dc60ad4094f18b51c4181324e54d3057524d6101 bitcoin-2ecaf214331b-win64-debug.zip 5b5a236b63e67f5f6c07ad9aa716aa7b72fb63722c96798b332c6d164738f9cf bitcoin-2ecaf214331b-win64-setup-unsigned.exe c1fa5894c5e02a201637567c80b9bde9024f44673dcd06fd4d489c1709179279 bitcoin-2ecaf214331b-win64.zip b40ba2d5da498330ade92a4ccebcceb1452b94c8ffeacb336f87e93b5c88d8af src/bitcoin-2ecaf214331b.tar.gz 665fd7eb61aed368150db58a254f15fb5efb51a4efa5abcc52571cb7a1a5de22 bitcoin-core-win-22-res.yml ``` #### macOS ```bash 6a1deae7662aa782baa82a42590f862c6bcdc4f4e38daa9b8c2a9eed1fbb5397 bitcoin-2ecaf214331b-osx-unsigned.dmg 1ee843266e84928a4323fa255c833528c2617a2c9fd2f98fb26ba19bbfc1227b bitcoin-2ecaf214331b-osx-unsigned.tar.gz 097b64dadc167d8e5b733421bf1541a40760ad952990f7cf3f35adc6ae2616d0 bitcoin-2ecaf214331b-osx64.tar.gz b40ba2d5da498330ade92a4ccebcceb1452b94c8ffeacb336f87e93b5c88d8af src/bitcoin-2ecaf214331b.tar.gz 6e378fb543928e40c7119b96be6ff773d38506a9a888f8b02c7f1b8a0801a80e bitcoin-core-osx-22-res.yml ``` ACKs for top commit: laanwj: Build script changes review ACK 2ecaf21 Tree-SHA512: 975d5830b787d2e08988f43cbc6e839294171c1d94c8219636308b05f9b77041421612ae67be24a631674670cfc9c2d96d8177f2b3158a78fc3deea19631febf

dongcarl · 2021-02-09T18:58:12Z

Python nerds: Anyone know why the security-check.py scripts can import pixie but test-security-check.py cannot import a function from my utils.py file?

maflcko · 2021-02-09T19:03:38Z

Does this help?

diff --git a/Makefile.am b/Makefile.am
index f6b824faaa..aed44113b8 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -58,6 +58,7 @@ DIST_SHARE = \
 
 BIN_CHECKS=$(top_srcdir)/contrib/devtools/symbol-check.py \
            $(top_srcdir)/contrib/devtools/security-check.py \
+           $(top_srcdir)/contrib/devtools/utils.py \
            $(top_srcdir)/contrib/devtools/pixie.py
 
 WINDOWS_PACKAGING = $(top_srcdir)/share/pixmaps/bitcoin.ico \

dongcarl · 2021-02-23T03:09:32Z

Pushed 809e14a → cae518c

Rebased on top of master

dongcarl · 2021-02-23T17:09:57Z

I tried running a build, but it seems to break due to the introduction of: #21255

Logs:

CC='x86_64-linux-gnu-gcc' CPPFILT=/gnu/store/3rjpkl6g8iwjis5rrpmgrblk21vz7pgx-profile/bin/x86_64-linux-gnu-c++filt /gnu/store/3rjpkl6g8iwjis5rrpmgrblk21vz7pgx-profile/bin/python3.8 ./contrib/devtools/test-symbol-check.py TestSymbolChecks.test_ELF
x86_64-linux-gnu-ld: /tmp/cczw0TSm.o: in function `main':
test1.c:(.text+0x1f): undefined reference to `renameat2'
collect2: error: ld returned 1 exit status
E
======================================================================
ERROR: test_ELF (__main__.TestSymbolChecks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./contrib/devtools/test-symbol-check.py", line 47, in test_ELF
    self.assertEqual(call_symbol_check(cc, source, executable, []),
  File "./contrib/devtools/test-symbol-check.py", line 15, in call_symbol_check
    subprocess.run([*cc,source,'-o',executable] + options, check=True)
  File "/gnu/store/jki2m0s42hzjfppdqdc7j3y4qlzawcl0-python-3.8.2/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['x86_64-linux-gnu-gcc', 'test1.c', '-o', 'test1']' returned non-zero exit status 1.

----------------------------------------------------------------------
Ran 1 test in 0.091s

FAILED (errors=1)
make: *** [Makefile:1439: test-security-check] Error 1

I think I know why: I constructed all of the guix cross-compilation toolchain to be glibc 2.27 based, which means that instead of the symbol check failing, it'll straight up not compile... Not sure what to do here, thoughts? @fanquake

fanquake · 2021-02-23T23:51:16Z

Spoke with Carl and the solution here is to move the Guix cross-compilation toolchain to be glibc 2.31 based.

c33b199 guix: Bump glibc and linux-headers (Carl Dong) 65363a1 guix: Rebase on 95aca2991b (1.2.0-12.dffc918) (Carl Dong) Pull request description: On bumping the time-machine: ``` A few changes which are useful for us: 1. 'gnu: cross-gcc-arguments: Enable 128 bit long double for POWER9.' is now merged into master. 2. gnutls is bumped to 3.6.15 and the temporal test failure in status-request-revoked is fixed. Note that this does not fix the case where one has installed Guix v1.2.0 and is running a substitute-less bootstrap build, since the `guix time-machine` command itself has a dependency on gnutls v3.6.12 (the one with the broken test) and will thus try to build it before attempting to jump forwards in time. This does however, mean that those who build a version of Guix that also contains this fix will not go backwards in time to build the broken gnutls v3.6.12. ``` On bumping the rest: ``` Bump glibc and linux-headers to match those of our Gitian counterparts. We also require a glibc >= 2.28 for the test-symbol-check scripts to work properly. The default BASE-GCC-FOR-LIBC also has to be bumped since glibc 2.31 requires a gcc >= 6.2 ``` This is a prerequisite for #20980 ACKs for top commit: fanquake: ACK c33b199 - I think going ahead with this now and to sycn back up to gitian is fine. It will also unblock #20980. Potential code signing related issues can be sorted out in #21239 and later PRs. Tree-SHA512: 31f022aadb93ba44813b0da005b1f2e5d67d76e8cdcdb53368924d1ea6cb076a21218c26831a6b0dcdcfe33507f54934330489ba557371d740f5587b7d727b95

dongcarl · 2021-03-02T22:49:45Z

Pushed cae518c -> db6e91a

Rebased over master
Use binutils disable flag patch from debian upstream for binutils 2.34
Add commit to use/test --reloc-section

b2dc314c882ba3dd119c44ed2673b1efe759f94ffd09a1f303a3bedc111cd39c  output/bitcoin-db6e91a5cdbd-aarch64-linux-gnu-debug.tar.gz
a0ae4738bd48c9cad43e4d45cfd3247462a96f1a2558bd27f81a7c44a8213883  output/bitcoin-db6e91a5cdbd-aarch64-linux-gnu.tar.gz
7b189a772cf0eb0911f137780b16c6e3bf12cd7663f7c03be03b4450797210dd  output/bitcoin-db6e91a5cdbd-arm-linux-gnueabihf-debug.tar.gz
40d3a6255484761e899a9ce75c35b0bb03d0612a6b80b27ac3910e28e5ca48dd  output/bitcoin-db6e91a5cdbd-arm-linux-gnueabihf.tar.gz
e70e92ce37132641b66a99a53716b3b66e61e0b096ecfee6d321a56a64a850d3  output/bitcoin-db6e91a5cdbd-osx-unsigned.dmg
d5ea424fd1083878e95cb6b7c09a6b0b3e716b8a0a6e37c403864fe99ec9477a  output/bitcoin-db6e91a5cdbd-osx-unsigned.tar.gz
8254778671c315aec66dbcfc020ff19bbf6070d61a1bb5d5880ac3e3c3ef8681  output/bitcoin-db6e91a5cdbd-osx64.tar.gz
a4dafcf884c89fca24109946e66f5c411d1f81154279ac1013a0a69d41b2650f  output/bitcoin-db6e91a5cdbd-powerpc64-linux-gnu-debug.tar.gz
02934a669612312b461e3d66623bfeeeb17088a173650db15ff8fc52eec529c4  output/bitcoin-db6e91a5cdbd-powerpc64-linux-gnu.tar.gz
f7de27d951003d632dd19447c13b96e575759380d1d15fb0c1d7272cc963b074  output/bitcoin-db6e91a5cdbd-powerpc64le-linux-gnu-debug.tar.gz
16a6cc048e04ea59e58855cdfb9fa653eab941e47e1edf4c4abc12edadd25b7d  output/bitcoin-db6e91a5cdbd-powerpc64le-linux-gnu.tar.gz
4a72908757e2ea4e9d5c9051b92e7cd10ab7193cae902d1631c364e78b03810b  output/bitcoin-db6e91a5cdbd-riscv64-linux-gnu-debug.tar.gz
3a26eecf0da5ed66c8fad9d13ffd342f2d7492878cbf3699056d1f3ebae3ee43  output/bitcoin-db6e91a5cdbd-riscv64-linux-gnu.tar.gz
99da16a244e8711c8b3a340c71b40f4f41248410629060d2fa59b2366cf7a41b  output/bitcoin-db6e91a5cdbd-win-unsigned.tar.gz
214c02c42f0932c988c0112762e9ee55e66b697a68fd22109aec89478b88a8ab  output/bitcoin-db6e91a5cdbd-win64-debug.zip
9b9810ed2fe6cf74d134618fcb6661184d8025902d82f4f8c10bd920a0c32e26  output/bitcoin-db6e91a5cdbd-win64-setup-unsigned.exe
059722aa12c33aec749c64a2a6ae395ebc4f62fe6e76d5c36c82437b17d1b627  output/bitcoin-db6e91a5cdbd-win64.zip
658256d2594448715944463d345cb3b3db3f55e1d5152dbfcaa278b247cd0fcc  output/bitcoin-db6e91a5cdbd-x86_64-linux-gnu-debug.tar.gz
bf8c5725bd2b475172dcda7d30d8e283612a776115101429753ade9ae5085576  output/bitcoin-db6e91a5cdbd-x86_64-linux-gnu.tar.gz
5020065aef12af03f056cb2810a74a646618a3d0b98a49e0a48d98808d8616e1  output/src/bitcoin-db6e91a5cdbd.tar.gz

DrahtBot · 2021-03-03T10:06:47Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

Erlay: bandwidth-efficient transaction relay protocol #21515 by naumenkogs

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

fanquake

I think this looks pretty good now. Going to run some builds.

Can you exclude contrib/guix/patches/ from codespell so we don't have to deal with this:

contrib/guix/patches/binutils-mingw-w64-disable-flags.patch:61: SEH ==> SHE
contrib/guix/patches/binutils-mingw-w64-disable-flags.patch:145: SEH ==> SHE
^ Warning: codespell identified likely spelling errors. Any false positives? Add them to the list of ignored words in test/lint/lint-spelling.ignore-words.txt

diff --git a/test/lint/lint-spelling.sh b/test/lint/lint-spelling.sh
index fbdf3c59c..238fa63c4 100755
--- a/test/lint/lint-spelling.sh
+++ b/test/lint/lint-spelling.sh
@@ -15,6 +15,6 @@ if ! command -v codespell > /dev/null; then
 fi
 
 IGNORE_WORDS_FILE=test/lint/lint-spelling.ignore-words.txt
-if ! codespell --check-filenames --disable-colors --quiet-level=7 --ignore-words=${IGNORE_WORDS_FILE} $(git ls-files -- ":(exclude)build-aux/m4/" ":(exclude)contrib/seeds/*.txt" ":(exclude)depends/" ":(exclude)doc/release-notes/" ":(exclude)src/leveldb/" ":(exclude)src/crc32c/" ":(exclude)src/qt/locale/" ":(exclude)src/qt/*.qrc" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/" ":(exclude)contrib/gitian-keys/keys.txt"); then
+if ! codespell --check-filenames --disable-colors --quiet-level=7 --ignore-words=${IGNORE_WORDS_FILE} $(git ls-files -- ":(exclude)build-aux/m4/" ":(exclude)contrib/seeds/*.txt" ":(exclude)depends/" ":(exclude)doc/release-notes/" ":(exclude)src/leveldb/" ":(exclude)src/crc32c/" ":(exclude)src/qt/locale/" ":(exclude)src/qt/*.qrc" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/" ":(exclude)contrib/gitian-keys/keys.txt" ":(exclude)contrib/guix/patches"); then
     echo "^ Warning: codespell identified likely spelling errors. Any false positives? Add them to the list of ignored words in ${IGNORE_WORDS_FILE}"

fanquake · 2021-03-04T01:50:01Z

configure.ac

@@ -884,6 +884,7 @@ if test x$use_hardening != xno; then
    ])
  fi

+  AX_CHECK_LINK_FLAG([[-Wl,--enable-reloc-section]], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,--enable-reloc-section"],, [[$LDFLAG_WERROR]])


In d3e6ee6. I think testing for this, and adding to our hardened ldflags when available is fine. It's enabled by default, however we like to be explicit. It's also available with the binutils (2.34) we are using for gitian builds.

Note that some of these flags also imply each other:
--high-entropy-va implies --dynamic-base & --enable-reloc-section
--dynamic-base implies --enable-reloc-section
``

fanquake · 2021-03-04T02:00:24Z

contrib/guix/patches/binutils-mingw-w64-disable-flags.patch

@@ -0,0 +1,171 @@
+Description: Add disable opposites to the security-related flags


Checked that this matches https://salsa.debian.org/mingw-w64-team/binutils-mingw-w64/-/blob/master/debian/patches/disable-flags.patch bar whitespace changes.

fanquake · 2021-03-04T02:04:36Z

contrib/devtools/test-security-check.py

        write_testcode(source)

-        self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--no-nxcompat','-Wl,--no-dynamicbase','-Wl,--no-high-entropy-va','-no-pie','-fno-PIE']),
+        self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--no-nxcompat','-Wl,--disable-reloc-section','-Wl,--no-dynamicbase','-Wl,--no-high-entropy-va','-no-pie','-fno-PIE']),


At this stage we have already given in to not being able to run the test security check target for windows in gitian due to lack of --no options in ld, so adding --disable here to test --enable-reloc-section isn't making anything worse. If anything this speaks to the usefulness of Guix, given how easy it is to patch these --no/--disable flags back into our toolchain. It would be much more difficult trying to achieve the same using gitian.

fanquake · 2021-03-04T03:56:14Z

One transient failure while building:

substitution of /gnu/store/04qddg51ih327yc8p7q2vn00slg4v1n9-gcc-cross-x86_64-w64-mingw32-9.3.0-lib complete
 binutils-cross-x86_64-w64-mingw32-2.34                                            18.4MiB/s 00:01 | 27.1MiB transferred

downloading from https://guix.carldong.io/nar/gzip/r7kbdcmb1w4is2bwjxx8jqy9fpb9pa9b-ld-wrapper-x86_64-w64-mingw32-0 ...
 ld-wrapper-x86_64-w64-mingw32-0                                                      6.1MiB/s 00:00 | 19KiB transferred

Backtrace:
In guix/ui.scm:
  2164:12 19 (run-guix-command _ . _)
In guix/scripts/substitute.scm:
    931:2 18 (guix-substitute . _)
In unknown file:
          17 (with-continuation-barrier #<procedure thunk ()>)
In ice-9/boot-9.scm:
  1736:10 16 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          15 (apply-smob/0 #<thunk 7f1a2032fdc0>)
In ice-9/boot-9.scm:
  1736:10 14 (with-exception-handler _ _ #:unwind? _ # _)
  1736:10 13 (with-exception-handler _ _ #:unwind? _ # _)
  1731:15 12 (with-exception-handler #<procedure 7f1a1dd610f0 at ic?> ?)
In guix/scripts/substitute.scm:
   980:17 11 (_)
    689:7 10 (process-substitution _ "/gnu/store/grb2m42291nkny2vid?" ?)
In ice-9/boot-9.scm:
  1736:10  9 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
    698:9  8 (_)
In ice-9/boot-9.scm:
  1731:15  7 (with-exception-handler #<procedure 7f1a1ecd18a0 at ic?> ?)
  1669:16  6 (raise-exception _ #:continuable? _)
  1667:16  5 (raise-exception _ #:continuable? _)
  1669:16  4 (raise-exception _ #:continuable? _)
  1764:13  3 (_ #<&compound-exception components: (#<&error> #<&irri?>)
  1669:16  2 (raise-exception _ #:continuable? _)
  1667:16  1 (raise-exception _ #:continuable? _)
  1669:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1669:16: In procedure raise-exception:
Bad http-version header component: K?%s-??

Backtrace:
           1 (primitive-load "/gnu/store/lvp5s8l0zwkrn2a0mmh6wf6z9ja?")
In guix/ui.scm:
  2164:12  0 (run-guix-command _ . _)

guix/ui.scm:2164:12: In procedure run-guix-command:
Bad http-version header component: K?%s-??

substitution of /gnu/store/grb2m42291nkny2vid35w7xrgirkxnrk-gcc-cross-x86_64-w64-mingw32-9.3.0 failed
guix environment: error: some substitutes for the outputs of derivation `/gnu/store/71f0wbcm7v2kbs3jfxjp44a7gx2iz66q-gcc-cross-x86_64-w64-mingw32-9.3.0.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source

but it looks like I've got matches except for output/bitcoin-db6e91a5cdbd-osx-unsigned.tar.gz :

find output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
b2dc314c882ba3dd119c44ed2673b1efe759f94ffd09a1f303a3bedc111cd39c  output/bitcoin-db6e91a5cdbd-aarch64-linux-gnu-debug.tar.gz
a0ae4738bd48c9cad43e4d45cfd3247462a96f1a2558bd27f81a7c44a8213883  output/bitcoin-db6e91a5cdbd-aarch64-linux-gnu.tar.gz
7b189a772cf0eb0911f137780b16c6e3bf12cd7663f7c03be03b4450797210dd  output/bitcoin-db6e91a5cdbd-arm-linux-gnueabihf-debug.tar.gz
40d3a6255484761e899a9ce75c35b0bb03d0612a6b80b27ac3910e28e5ca48dd  output/bitcoin-db6e91a5cdbd-arm-linux-gnueabihf.tar.gz
e70e92ce37132641b66a99a53716b3b66e61e0b096ecfee6d321a56a64a850d3  output/bitcoin-db6e91a5cdbd-osx-unsigned.dmg
18b8f49e36a35f7caeb7e2c34410884bd9e20e3dd4c875afe7202610918c1084  output/bitcoin-db6e91a5cdbd-osx-unsigned.tar.gz
8254778671c315aec66dbcfc020ff19bbf6070d61a1bb5d5880ac3e3c3ef8681  output/bitcoin-db6e91a5cdbd-osx64.tar.gz
a4dafcf884c89fca24109946e66f5c411d1f81154279ac1013a0a69d41b2650f  output/bitcoin-db6e91a5cdbd-powerpc64-linux-gnu-debug.tar.gz
02934a669612312b461e3d66623bfeeeb17088a173650db15ff8fc52eec529c4  output/bitcoin-db6e91a5cdbd-powerpc64-linux-gnu.tar.gz
f7de27d951003d632dd19447c13b96e575759380d1d15fb0c1d7272cc963b074  output/bitcoin-db6e91a5cdbd-powerpc64le-linux-gnu-debug.tar.gz
16a6cc048e04ea59e58855cdfb9fa653eab941e47e1edf4c4abc12edadd25b7d  output/bitcoin-db6e91a5cdbd-powerpc64le-linux-gnu.tar.gz
4a72908757e2ea4e9d5c9051b92e7cd10ab7193cae902d1631c364e78b03810b  output/bitcoin-db6e91a5cdbd-riscv64-linux-gnu-debug.tar.gz
3a26eecf0da5ed66c8fad9d13ffd342f2d7492878cbf3699056d1f3ebae3ee43  output/bitcoin-db6e91a5cdbd-riscv64-linux-gnu.tar.gz
99da16a244e8711c8b3a340c71b40f4f41248410629060d2fa59b2366cf7a41b  output/bitcoin-db6e91a5cdbd-win-unsigned.tar.gz
214c02c42f0932c988c0112762e9ee55e66b697a68fd22109aec89478b88a8ab  output/bitcoin-db6e91a5cdbd-win64-debug.zip
9b9810ed2fe6cf74d134618fcb6661184d8025902d82f4f8c10bd920a0c32e26  output/bitcoin-db6e91a5cdbd-win64-setup-unsigned.exe
059722aa12c33aec749c64a2a6ae395ebc4f62fe6e76d5c36c82437b17d1b627  output/bitcoin-db6e91a5cdbd-win64.zip
658256d2594448715944463d345cb3b3db3f55e1d5152dbfcaa278b247cd0fcc  output/bitcoin-db6e91a5cdbd-x86_64-linux-gnu-debug.tar.gz
bf8c5725bd2b475172dcda7d30d8e283612a776115101429753ade9ae5085576  output/bitcoin-db6e91a5cdbd-x86_64-linux-gnu.tar.gz
5020065aef12af03f056cb2810a74a646618a3d0b98a49e0a48d98808d8616e1  output/src/bitcoin-db6e91a5cdbd.tar.gz

dongcarl · 2021-05-07T19:34:35Z

Pushed 69c650b...4914a08

Rebased over master now that contrib: use LIEF for macOS and Windows symbol & security checks #21664 is merged.

hebasto · 2021-05-09T12:41:00Z

https://cirrus-ci.com/task/5154764032835584?logs=lint#L855

contrib/devtools/symbol-check.py:15:1: F401 'os' imported but unused
contrib/devtools/test-security-check.py:10:1: F401 'typing.List' imported but unused
Success: no issues found in 201 source files
^---- failure generated from test/lint/lint-python.sh

hebasto

Approach ACK 4914a08

The commit 8a833f3 "devtools: Improve *-check.py tool detection" is broken without the 777eae3 "devtools: Pass make $(CC) into test-*-check.py":

$ test/lint/lint-python.sh 
contrib/devtools/test-security-check.py:12:1: F401 'utils.determine_wellknown_cmd' imported but unused
contrib/devtools/test-symbol-check.py:12:1: F401 'utils.determine_wellknown_cmd' imported but unused
contrib/devtools/test-symbol-check.py:14:27: F821 undefined name 'List'
contrib/devtools/test-symbol-check.py:14: error: Name 'List' is not defined
contrib/devtools/test-symbol-check.py:14: note: Did you forget to import it from "typing"? (Suggestion: "from typing import List")
Found 1 error in 1 file (checked 201 source files)

Maybe combine them, or reorder changes?

hebasto · 2021-05-09T13:08:46Z

configure.ac

@@ -911,6 +911,7 @@ if test x$use_hardening != xno; then
    ])
  fi

+  AX_CHECK_LINK_FLAG([[-Wl,--enable-reloc-section]], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,--enable-reloc-section"],, [[$LDFLAG_WERROR]])


style nit: I know it follows the surrounding style, but the double quoting is really unneeded here:

Suggested change

AX_CHECK_LINK_FLAG([[-Wl,--enable-reloc-section]], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,--enable-reloc-section"],, [[$LDFLAG_WERROR]])

AX_CHECK_LINK_FLAG([-Wl,--enable-reloc-section], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,--enable-reloc-section"], [], [$LDFLAG_WERROR])

When using mypy ignore directives, the error code needs to be specified. Somehow mypy doesn't print it by default...

This is important to make sure that we're not testing tools different from the one we're building with. Introduce determine_wellknown_cmd, which encapsulates how we should handle well-known tools specification (IFS splitting, env override, etc.).

We use these flags in our test-security-check make target, but they are only available because debian patches them in. We can patch them in for our Guix builds so that we can check the sanity of our security/symbol checking suite before running them.

Also fix test-security-check.py to account for new PE PIE failure indication.

dongcarl · 2021-05-12T19:48:32Z

Pushed 4914a08 -> d9a3d32

Squashed a few commits which no longer make sense to be separate
Rebased over master
Addressed: guix: Test security-check sanity before performing them #20980 (review)

hebasto · 2021-05-13T09:33:11Z

Testing this PR together with #21871 reveals some kind of incompatibility.

fanquake · 2021-07-01T08:43:05Z

I am fixing the macOS issues with test-security-check, so this can be part of 22.0.

fanquake · 2021-07-01T13:03:20Z

I've opened #22381, which is a rebased version of this, with the macOS fix.

…ng them (with macOS) 5b4703c guix: Test security-check sanity before performing them (Carl Dong) 6cf3345 scripts: adjust test-symbol-check for guix release environment (fanquake) 1946b5f scripts: more robustly test macOS symbol checks (fanquake) a8127b3 build: Use and test PE binutils with --reloc-section (Carl Dong) 678348d guix: Patch binutils to add security-related disable flags (Carl Dong) 9fdc8af devtools: Improve *-check.py tool detection (Carl Dong) bda62ea ci: skip running the Linux test-security-check target for now (fanquake) d6ef354 lint: Run mypy with --show-error-codes (Carl Dong) Pull request description: This is bitcoin#20980 rebased (to include the Boost Process fix), and with an additional commit (892d689) to fix running the `test-security-check` target for the macOS build. It should pass inside Guix, as well as when cross-compiling on Ubuntu, or building natively on macOS. Note that the `test-security-check` may output some warnings (similar too): ```bash ld: warning: passed two min versions (10.14, 11.4) for platform macOS. Using 11.4. ld: warning: passed two min versions (10.14, 11.4) for platform macOS. Using 11.4. ld: warning: passed two min versions (10.14, 10.14) for platform macOS. Using 10.14. ``` but those can be ignored, and come about due to us passing `-platform_version` when `-mmacosx-version-min` is already part of `CC`. Guix builds: ```bash 71ed0c7a13a4726300779ffc87f7d271086a2744c36896fe6dc51fe3dc33df2e guix-build-5b4703c6a70d/output/aarch64-linux-gnu/SHA256SUMS.part 9273980a17052c8ec45b77579781c14ab5d189fa25aa29907d5115513dd302b1 guix-build-5b4703c6a70d/output/aarch64-linux-gnu/bitcoin-5b4703c6a70d-aarch64-linux-gnu-debug.tar.gz 9c042179af43c8896eb95a34294df15d4910308dcdba40b2010cd36e192938b8 guix-build-5b4703c6a70d/output/aarch64-linux-gnu/bitcoin-5b4703c6a70d-aarch64-linux-gnu.tar.gz 1ceddecac113f50a952ba6a201cdcdb722e3dc804e663f219bfac8268ce42bf0 guix-build-5b4703c6a70d/output/arm-linux-gnueabihf/SHA256SUMS.part 759597c4e925e75db4a2381c06cda9b9f4e4674c23436148676b31c9be05c7aa guix-build-5b4703c6a70d/output/arm-linux-gnueabihf/bitcoin-5b4703c6a70d-arm-linux-gnueabihf-debug.tar.gz 34e3b6beabaf8c95d7c2ca0d2c3ac4411766694ef43e00bd9783badbbaf045a7 guix-build-5b4703c6a70d/output/arm-linux-gnueabihf/bitcoin-5b4703c6a70d-arm-linux-gnueabihf.tar.gz e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 guix-build-5b4703c6a70d/output/dist-archive/SKIPATTEST.TAG 3664f6ceee7898caa374281fd877a7597fe491fa2e9f0c174c28d889d60b559c guix-build-5b4703c6a70d/output/dist-archive/bitcoin-5b4703c6a70d.tar.gz d6bc35ba0750c1440bb32831b8c12cddee62f6dce10fec2650897444c2bf4748 guix-build-5b4703c6a70d/output/powerpc64-linux-gnu/SHA256SUMS.part a836edf6474ba0c16c19bb217549bac7936c1b44306ed512df58f607ee5568f2 guix-build-5b4703c6a70d/output/powerpc64-linux-gnu/bitcoin-5b4703c6a70d-powerpc64-linux-gnu-debug.tar.gz 7cc91c6805d5069ca3bd1771e77d95f83eb184b137198cbf84d1d11d0a5c5afe guix-build-5b4703c6a70d/output/powerpc64-linux-gnu/bitcoin-5b4703c6a70d-powerpc64-linux-gnu.tar.gz 93b4cb7b83c4975120ad5de5a92f050f5760a2a3f2c37c204c647f5a581c924a guix-build-5b4703c6a70d/output/powerpc64le-linux-gnu/SHA256SUMS.part 2266e2c5d0dafa28c6c057ccfc1c439baeab1d714d8c3f64a83015d2827116d2 guix-build-5b4703c6a70d/output/powerpc64le-linux-gnu/bitcoin-5b4703c6a70d-powerpc64le-linux-gnu-debug.tar.gz 85f41f42c319b83d049d6fd2e2278c07b40a1e28a2eac596427822c0eef9dc3f guix-build-5b4703c6a70d/output/powerpc64le-linux-gnu/bitcoin-5b4703c6a70d-powerpc64le-linux-gnu.tar.gz 1499ca9119926083d8c3714ca10d8d4c8d864cbeee8848fd8445b7a1d081222d guix-build-5b4703c6a70d/output/riscv64-linux-gnu/SHA256SUMS.part 1995fc1a2e45c49d4b0718aff5dcdac931917e8ae9e762fd23f1126abcecc248 guix-build-5b4703c6a70d/output/riscv64-linux-gnu/bitcoin-5b4703c6a70d-riscv64-linux-gnu-debug.tar.gz 266889eb58429a470f0fd7bb123f2ae09b0aef86c47b0390938b3634a8f748a9 guix-build-5b4703c6a70d/output/riscv64-linux-gnu/bitcoin-5b4703c6a70d-riscv64-linux-gnu.tar.gz cdc3a0dcf80b110443dac5ddf8bc951001a776a651c898c5ea49bb2d487bfe29 guix-build-5b4703c6a70d/output/x86_64-apple-darwin18/SHA256SUMS.part 8538d1eab96c97866b24546c453d95822f24cf9c6638b42ba523eb7aa441cb26 guix-build-5b4703c6a70d/output/x86_64-apple-darwin18/bitcoin-5b4703c6a70d-osx-unsigned.dmg d1b73133f1da68586b07292a8425f7f851e93f599c016376f23728c041cf39cc guix-build-5b4703c6a70d/output/x86_64-apple-darwin18/bitcoin-5b4703c6a70d-osx-unsigned.tar.gz 5ad94c5f8a5f29405955ff3ab35d137de1acc04398d6c8298fb187b57a6e316a guix-build-5b4703c6a70d/output/x86_64-apple-darwin18/bitcoin-5b4703c6a70d-osx64.tar.gz 8c6d7b3f847faa7b4d16ceecf228f26f146ea982615c1d7a00c57f9230a0c484 guix-build-5b4703c6a70d/output/x86_64-linux-gnu/SHA256SUMS.part d0a8c99750319ad8046cfa132a54e5c13a08351f94439ae9af0f8e5486c2c2ea guix-build-5b4703c6a70d/output/x86_64-linux-gnu/bitcoin-5b4703c6a70d-x86_64-linux-gnu-debug.tar.gz d816bb26dd4b0e309f2f576b1cccc6d78743fb2f357daad2da09bb1177330971 guix-build-5b4703c6a70d/output/x86_64-linux-gnu/bitcoin-5b4703c6a70d-x86_64-linux-gnu.tar.gz 65caaa7f648c7eab1eb82c3331a2ca25b8cd4fe41439de55604501e02571de55 guix-build-5b4703c6a70d/output/x86_64-w64-mingw32/SHA256SUMS.part 5bf6f7328cbceb0db22a2d7babb07b60cb6dcc19a6db84a1698589b7f5173a06 guix-build-5b4703c6a70d/output/x86_64-w64-mingw32/bitcoin-5b4703c6a70d-win-unsigned.tar.gz 7aabcb56115decef78d3797840b6e49dbc9b202d56f892490e92616fb06fec9e guix-build-5b4703c6a70d/output/x86_64-w64-mingw32/bitcoin-5b4703c6a70d-win64-debug.zip 2f369694648ff9dc5ca1261a1e5874b1c7408ccf2802f9caef56c1334e8a5b7c guix-build-5b4703c6a70d/output/x86_64-w64-mingw32/bitcoin-5b4703c6a70d-win64-setup-unsigned.exe 1c1f92513c4aad38419ff49a7b80bf10e6b1eca01ee8c5e3b2acd1768cf1e3d5 guix-build-5b4703c6a70d/output/x86_64-w64-mingw32/bitcoin-5b4703c6a70d-win64.zip ``` ACKs for top commit: hebasto: Approach ACK 5b4703c. Tree-SHA512: 2cd92a245ea64ef7176cf402a1fa5348a9421c30a4d30d01c950c48f6dcc15cf22ce69ffe1657be97e5fccc14bd933d64683c4439b695528ce3dc34d72dda927

2ecaf21 gitian: remove execstack workaround for ricv64 & powerpc64le (fanquake) 5baff2b build: use focal in gitian descriptors (fanquake) Pull request description: This PR changes the gitian descriptors to use Ubuntu Focal (20.04), over Bionic (18.04), moving from GCC 7.5 to GCC 8.4 for native Linux builds, mingw-w64 GCC 7.3 to mingw-w64 GCC 9.3 for Windows builds, while continuing to use GCC 8.4 for all cross builds and Clang 8.0.0 for macOS builds. It also drops the `-Wl,-z,noexecstack` workaround we've been using for the riscv64 and powerpc64le hosts, as it's no-longer needed. One new package is installed in the osx build, `libtinfo5`, as libtinfo5.so is required by our downloaded Clang 8. A bump to Focal will at least be required if we want to update to a newer Qt (5.15, bitcoin#19716) for 22.0, as we need a newer version of [`g++-mingw-w64`](https://packages.ubuntu.com/focal/g++-mingw-w64-x86-64) and the [`mingw-w64`](https://mingw-w64.org/doku.php) headers. This can still be done while continuing to use GCC 8.4 for Linux builds (see below), however the newer `g++-mingw-w64` will be based off of GCC 9.3. **Some considerations** GCC 9 is affected by bitcoin#20005 "memcmp with constants that contain zero bytes are broken in GCC", and the newer `g++-mingw-w64` will be based off of GCC 9.3. The `--no-*` variants of the Windows linker flags (i.e `--no-dynamicbase`) we use to [test our `security-check.py` script](https://github.com/bitcoin/bitcoin/blob/16b784d953365bb2d7ae65acd2b20a79ef8ba7b6/contrib/devtools/test-security-check.py#L53) are not patched into the mingw binutils in Focal (they have been re-added in Groovy (20.10)). This isn't currently an issue, however, we might add a call to `test-security-check` for Guix (bitcoin#20980), and if we wanted to do the same for gitian, it would not work. Note how it's quite "easy" for us to apply the `--no-*` variant patch to our Guix build; it would be quite a bit harder to do in Gitian. Gitian Builds @ 2ecaf21 #### Linux ```bash 8882ea78486fbae4fac574b9089eb1107c6372d0dd7dfcda4f0f930576f9d6c1 bitcoin-2ecaf214331b-aarch64-linux-gnu-debug.tar.gz 50a9e30943b4eee5163edff3331241e745ff32a2c4463c21a6fdc5986e2d0383 bitcoin-2ecaf214331b-aarch64-linux-gnu.tar.gz ec4e55a447fddf033fee33cd5f22bfeda3c3612f059194bcf6238859f7989d7a bitcoin-2ecaf214331b-arm-linux-gnueabihf-debug.tar.gz 444fe1b3b933c00bcbd4a9d86888cff3b61c1215b1debccd2843e842d1224777 bitcoin-2ecaf214331b-arm-linux-gnueabihf.tar.gz 88e486ff465980dc1a4aab9687d142ec6f727ed2c52cf539f69db2877dee83b2 bitcoin-2ecaf214331b-powerpc64-linux-gnu-debug.tar.gz 66144ac264c65cada9d86446e6026c85b04fb88198b8f41b42840f6031db3e6c bitcoin-2ecaf214331b-powerpc64-linux-gnu.tar.gz 34bcc13d78d929d575e34e77a6672f23ca7ea23230b28ec2eed563889352ba86 bitcoin-2ecaf214331b-powerpc64le-linux-gnu-debug.tar.gz b4c5f959664f3063df4330edfe343c17120eb6b556ee1c15c4aeb2c1c54ffd49 bitcoin-2ecaf214331b-powerpc64le-linux-gnu.tar.gz 918fa72ab6f6ebce4e9663c93f72fe26651c260477cbb54749f7eb61438b5cc1 bitcoin-2ecaf214331b-riscv64-linux-gnu-debug.tar.gz f704f9f8c053ffe37d854e2e81e0f4c0614c435dad7f5d82518c681b73a76ae6 bitcoin-2ecaf214331b-riscv64-linux-gnu.tar.gz b59e3a62f1df9d79f30e916b3c9655f654036fe3a420040c53acc8dd9f4162c5 bitcoin-2ecaf214331b-x86_64-linux-gnu-debug.tar.gz a4dc9ca877cc97544e65db11be38406d16f15d74fcdcd2318bb92474729bc60d bitcoin-2ecaf214331b-x86_64-linux-gnu.tar.gz b40ba2d5da498330ade92a4ccebcceb1452b94c8ffeacb336f87e93b5c88d8af src/bitcoin-2ecaf214331b.tar.gz af6ebc91147778e4e6705eade62608dde4d6e60522d79087fa9129bdb7c01199 bitcoin-core-linux-22-res.yml ``` #### Windows ```bash 121a3970a6911cb8c453b2ce37d03f6cbb43333e29db8fa516c68563fb367f43 bitcoin-2ecaf214331b-win-unsigned.tar.gz 6294e9efebe935092f9ba119dc60ad4094f18b51c4181324e54d3057524d6101 bitcoin-2ecaf214331b-win64-debug.zip 5b5a236b63e67f5f6c07ad9aa716aa7b72fb63722c96798b332c6d164738f9cf bitcoin-2ecaf214331b-win64-setup-unsigned.exe c1fa5894c5e02a201637567c80b9bde9024f44673dcd06fd4d489c1709179279 bitcoin-2ecaf214331b-win64.zip b40ba2d5da498330ade92a4ccebcceb1452b94c8ffeacb336f87e93b5c88d8af src/bitcoin-2ecaf214331b.tar.gz 665fd7eb61aed368150db58a254f15fb5efb51a4efa5abcc52571cb7a1a5de22 bitcoin-core-win-22-res.yml ``` #### macOS ```bash 6a1deae7662aa782baa82a42590f862c6bcdc4f4e38daa9b8c2a9eed1fbb5397 bitcoin-2ecaf214331b-osx-unsigned.dmg 1ee843266e84928a4323fa255c833528c2617a2c9fd2f98fb26ba19bbfc1227b bitcoin-2ecaf214331b-osx-unsigned.tar.gz 097b64dadc167d8e5b733421bf1541a40760ad952990f7cf3f35adc6ae2616d0 bitcoin-2ecaf214331b-osx64.tar.gz b40ba2d5da498330ade92a4ccebcceb1452b94c8ffeacb336f87e93b5c88d8af src/bitcoin-2ecaf214331b.tar.gz 6e378fb543928e40c7119b96be6ff773d38506a9a888f8b02c7f1b8a0801a80e bitcoin-core-osx-22-res.yml ``` ACKs for top commit: laanwj: Build script changes review ACK 2ecaf21 Tree-SHA512: 975d5830b787d2e08988f43cbc6e839294171c1d94c8219636308b05f9b77041421612ae67be24a631674670cfc9c2d96d8177f2b3158a78fc3deea19631febf

dongcarl added Build system DrahtBot Guix build requested labels Jan 21, 2021

fanquake reviewed Jan 22, 2021

View reviewed changes

dongcarl force-pushed the 2020-12-guix-mingw-extra-flags branch from 460b697 to 178ebd0 Compare January 22, 2021 20:10

DrahtBot removed the DrahtBot Guix build requested label Jan 30, 2021

fanquake mentioned this pull request Jan 30, 2021

gitian: Bump descriptors to Focal for 22.0 #21036

Merged

dongcarl force-pushed the 2020-12-guix-mingw-extra-flags branch 2 times, most recently from 02a36e2 to e9bdb5b Compare February 5, 2021 20:22

dongcarl force-pushed the 2020-12-guix-mingw-extra-flags branch 4 times, most recently from 4ebaa36 to cae518c Compare February 23, 2021 03:07

dongcarl mentioned this pull request Feb 25, 2021

guix: Bump time-machine, glibc, and linux-headers #21298

Merged

dongcarl force-pushed the 2020-12-guix-mingw-extra-flags branch from cae518c to db6e91a Compare March 2, 2021 22:47

maflcko added the DrahtBot Guix build requested label Mar 3, 2021

DrahtBot mentioned this pull request Mar 3, 2021

scripts: check for control flow instrumentation in security-check.py #21135

Closed

fanquake reviewed Mar 4, 2021

View reviewed changes

DrahtBot mentioned this pull request May 4, 2021

ci: Enable D_GLIBCXX_DEBUG for multiprocess task #21812

Merged

DrahtBot added the Needs rebase label May 4, 2021

dongcarl force-pushed the 2020-12-guix-mingw-extra-flags branch from 69c650b to 4914a08 Compare May 7, 2021 19:34

DrahtBot removed the Needs rebase label May 7, 2021

DrahtBot mentioned this pull request May 7, 2021

Add minisketch subtree and integrate in build/test #21859

Closed

hebasto reviewed May 9, 2021

View reviewed changes

dongcarl force-pushed the 2020-12-guix-mingw-extra-flags branch from 4914a08 to 78ca6a8 Compare May 12, 2021 19:45

dongcarl added 5 commits May 12, 2021 15:47

lint: Run mypy with --show-error-codes

9114280

When using mypy ignore directives, the error code needs to be specified. Somehow mypy doesn't print it by default...

build: Use and test PE binutils with --reloc-section

d1d20ae

Also fix test-security-check.py to account for new PE PIE failure indication.

guix: Test security-check sanity before performing them

d9a3d32

dongcarl force-pushed the 2020-12-guix-mingw-extra-flags branch from 78ca6a8 to d9a3d32 Compare May 12, 2021 19:47

hebasto mentioned this pull request May 12, 2021

scripts: add checks for minimum required OS versions #21871

Merged

fanquake added this to the 22.0 milestone Jul 1, 2021

fanquake mentioned this pull request Jul 1, 2021

guix: Test security-check sanity before performing them (with macOS) #22381

Merged

fanquake closed this Jul 1, 2021

fanquake mentioned this pull request Jul 1, 2021

guix: Avoid relying on newer symbols by rebasing our cross toolchains on older glibcs #22365

Merged

bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022

		@@ -0,0 +1,171 @@
		Description: Add disable opposites to the security-related flags

	AX_CHECK_LINK_FLAG([[-Wl,--enable-reloc-section]], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,--enable-reloc-section"],, [[$LDFLAG_WERROR]])
	AX_CHECK_LINK_FLAG([-Wl,--enable-reloc-section], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,--enable-reloc-section"], [], [$LDFLAG_WERROR])

guix: Test security-check sanity before performing them #20980

guix: Test security-check sanity before performing them #20980

Uh oh!

Conversation

dongcarl commented Jan 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fanquake left a comment

Choose a reason for hiding this comment

Uh oh!

fanquake Jan 22, 2021

Choose a reason for hiding this comment

Uh oh!

dongcarl Jan 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

practicalswift commented Jan 26, 2021

Uh oh!

maflcko commented Jan 30, 2021

Uh oh!

dongcarl commented Feb 9, 2021

Uh oh!

maflcko commented Feb 9, 2021

Uh oh!

dongcarl commented Feb 23, 2021

Uh oh!

dongcarl commented Feb 23, 2021

Uh oh!

fanquake commented Feb 23, 2021

Uh oh!

dongcarl commented Mar 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Mar 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Conflicts

Uh oh!

fanquake left a comment

Choose a reason for hiding this comment

Uh oh!

fanquake Mar 4, 2021

Choose a reason for hiding this comment

Uh oh!

fanquake Mar 4, 2021

Choose a reason for hiding this comment

Uh oh!

fanquake Mar 4, 2021

Choose a reason for hiding this comment

Uh oh!

fanquake commented Mar 4, 2021

Uh oh!

dongcarl commented May 7, 2021

Uh oh!

hebasto commented May 9, 2021

Uh oh!

hebasto left a comment

Choose a reason for hiding this comment

Uh oh!

hebasto May 9, 2021

Choose a reason for hiding this comment

Uh oh!

dongcarl commented May 12, 2021

Uh oh!

hebasto commented May 13, 2021

Uh oh!

fanquake commented Jul 1, 2021

Uh oh!

fanquake commented Jul 1, 2021

Uh oh!

Uh oh!

dongcarl commented Jan 21, 2021 •

edited

Loading

dongcarl Jan 22, 2021 •

edited

Loading

dongcarl commented Mar 2, 2021 •

edited

Loading

DrahtBot commented Mar 3, 2021 •

edited

Loading