scripts: use LIEF for ELF security & symbol checks #22392
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
This was referenced Jul 2, 2021
|
Concept ACK |
|
Apologies if this information is elsewhere, but could you summarize the advantages, and disadvantages if any, of using LIEF over Wladimir's script? Does this add a required dependency to the build? Thank you. |
Bitcoin Core really shouldn't be maintaining it's own ELF parser. This has already proved somewhat problematic, i.e: #21454.
No. It's already needed for release builds, and used for the macOS and Windows binaries. |
|
Concept ACK |
|
Concept ACK.
Not adding a dependency was my reason for writing my own, at the time. But as now we already rely on LIEF for windows PE and OSX, using it for ELF too seems fair, it's better than relying on a hodge-podge of text output of various tools, and I'm definitely not planning to write PE and OSX binary parsers. (I wish I had known of LIEF sooner though, would have saved me some time) |
This was referenced Sep 10, 2021
dfccdbc to
725fd00
Compare
|
Rebased and un-drafted. Fixed up some outstanding issues in the security checks. This should be ready for review. Now also includes some of the changes from #23148. |
|
Guix build: bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
6c81cb5c88643da842fecd1466ab0112877f91ef5ca558fa11663c546c571a72 guix-build-63b63720bd05/output/aarch64-linux-gnu/SHA256SUMS.part
a1a96d8f4555a87e66b5806f43239f380f49d8107b5479265f7412ff303e5c4b guix-build-63b63720bd05/output/aarch64-linux-gnu/bitcoin-63b63720bd05-aarch64-linux-gnu-debug.tar.gz
436d1e596fbcb8a00c162a5484db1eed397398bf39af22d065261e53fbdab9a2 guix-build-63b63720bd05/output/aarch64-linux-gnu/bitcoin-63b63720bd05-aarch64-linux-gnu.tar.gz
fd3eda71820a2514007aac4662cb2f162038a8f9d0893c5b8754a2fc54f158c5 guix-build-63b63720bd05/output/arm-linux-gnueabihf/SHA256SUMS.part
61a6401763e0fdda2d046c01e5cc54c7c6e8dc636092c9f7dbfd37413496504e guix-build-63b63720bd05/output/arm-linux-gnueabihf/bitcoin-63b63720bd05-arm-linux-gnueabihf-debug.tar.gz
18ea82384511a9ea93ee53e520132ddb1d9cac174d22928babef269db7447723 guix-build-63b63720bd05/output/arm-linux-gnueabihf/bitcoin-63b63720bd05-arm-linux-gnueabihf.tar.gz
a34148a28bdbd05c5002fc319f6d57b2e854229ea6077d94178ccbdbad3eb538 guix-build-63b63720bd05/output/dist-archive/bitcoin-63b63720bd05.tar.gz
d0d3177e2fde017ce5286e0310ac75fbbf5a06f65aba05e89f2994eeae64c515 guix-build-63b63720bd05/output/powerpc64-linux-gnu/SHA256SUMS.part
fddd42264d5cf31920185328718a5310a9efec6dd6b8b127ad10098d41dae819 guix-build-63b63720bd05/output/powerpc64-linux-gnu/bitcoin-63b63720bd05-powerpc64-linux-gnu-debug.tar.gz
bc6d969c77e4b5bd8d117081a3a45752441a415fcaf2c6fcf50930794258e6a7 guix-build-63b63720bd05/output/powerpc64-linux-gnu/bitcoin-63b63720bd05-powerpc64-linux-gnu.tar.gz
3e49b0974709c6bc1df2c2e0887534cc82940dad6f37621f19f9712d64b8eb6f guix-build-63b63720bd05/output/powerpc64le-linux-gnu/SHA256SUMS.part
9e2eb5582074f079712f72fae6b7e65449372a1dc5bb6431ca5394b2d72d0001 guix-build-63b63720bd05/output/powerpc64le-linux-gnu/bitcoin-63b63720bd05-powerpc64le-linux-gnu-debug.tar.gz
1280a0a97653f02da39f91073cd7e65ac402da60546a20e5dd0c0cd8bf27b76c guix-build-63b63720bd05/output/powerpc64le-linux-gnu/bitcoin-63b63720bd05-powerpc64le-linux-gnu.tar.gz
a8fe290b29e5eca0fcdc4988d730c66e5a02df7fa0a74a9e2a38c9624ed4ad6f guix-build-63b63720bd05/output/riscv64-linux-gnu/SHA256SUMS.part
b04449be302f6fb5010ad7a221be6c5980a162fa0db5b208d18028a5002acf7f guix-build-63b63720bd05/output/riscv64-linux-gnu/bitcoin-63b63720bd05-riscv64-linux-gnu-debug.tar.gz
427912609cc80024a093c41c283948905b15afc6da3279a56f414646bd61d805 guix-build-63b63720bd05/output/riscv64-linux-gnu/bitcoin-63b63720bd05-riscv64-linux-gnu.tar.gz
1157360f2c145449e0bc2796dd92d57e0cc7e6735553786cf9ce8285f8b333f1 guix-build-63b63720bd05/output/x86_64-apple-darwin19/SHA256SUMS.part
55032d0ad298e276d61fbe6976dfd4173f32597fce5c8099b1da359f101cc5fe guix-build-63b63720bd05/output/x86_64-apple-darwin19/bitcoin-63b63720bd05-osx-unsigned.dmg
af4fd797f626008464dba1a03ff70c5be24a163bd2685870242308d71113832c guix-build-63b63720bd05/output/x86_64-apple-darwin19/bitcoin-63b63720bd05-osx-unsigned.tar.gz
b2a4f43d1da25caf047c05663611bd2521faf68a43fac4970480a00f3c52d648 guix-build-63b63720bd05/output/x86_64-apple-darwin19/bitcoin-63b63720bd05-osx64.tar.gz
7f696f6750b376f832a70912b270425c9fb54ce339d3f5afb808ff6e2a3ee224 guix-build-63b63720bd05/output/x86_64-linux-gnu/SHA256SUMS.part
52776b3b2b8d3945c6cc16bcbc1574ba60adeea141f8a76614b6c39dde1d5614 guix-build-63b63720bd05/output/x86_64-linux-gnu/bitcoin-63b63720bd05-x86_64-linux-gnu-debug.tar.gz
9df46df15f1fd2c1a619c23d550434a37ac7fd2c845c831957d7cc1d8b90b072 guix-build-63b63720bd05/output/x86_64-linux-gnu/bitcoin-63b63720bd05-x86_64-linux-gnu.tar.gz
f3f297b6e17c75e9b0bcfb0100fb678166617f13b29f53de521b2f12c8fcd3f8 guix-build-63b63720bd05/output/x86_64-w64-mingw32/SHA256SUMS.part
ef0bffde3a0de16b40812f65ecf2494d4ae555c56aa613c1bc66cbb983471a1f guix-build-63b63720bd05/output/x86_64-w64-mingw32/bitcoin-63b63720bd05-win-unsigned.tar.gz
58c930bc308c87b01049704b723520fddba5273e3dc2b1b4b8cee3e70c803ea6 guix-build-63b63720bd05/output/x86_64-w64-mingw32/bitcoin-63b63720bd05-win64-debug.zip
c6eb7174b9dfa516089a29d0780b2d649c4b8d4bf3203b7f05e6bb720b9ceb38 guix-build-63b63720bd05/output/x86_64-w64-mingw32/bitcoin-63b63720bd05-win64-setup-unsigned.exe
1b0f8a91a86bf16396dbf8afd014e9ac4b62cea5fa0024680ed5f2d1ea5bd5b5 guix-build-63b63720bd05/output/x86_64-w64-mingw32/bitcoin-63b63720bd05-win64.zip |
Guix builds: |
laanwj
reviewed
Oct 11, 2021
laanwj
reviewed
Oct 11, 2021
These test-*-check scripts should compile "test" binaries in a way that is as close to what autotools would do, since the goal is to make sure that if we run the *-check script, they can correctly detect flaws in binaries which are compiled by our autotools-based system. Therefore, we should emulate what happens when the binary is linked in autotools, meaning that for C binaries, we need to supply the CFLAGS, CPPFLAGS, and LDFLAGS flags in that order. Note to future developers: perhaps it'd be nice to have these test-*-check scripts be part of configure.ac to avoid having to manually replicate autoconf-like behaviour every time we find a discrepancy. Of course, that would also mean you'd have to write more m4...
Co-authored-by: Carl Dong <contact@carldong.me>
|
Guix build: bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
78269d3163b9898f1853887e2303ad3987c3c6720c53919f3c5ad51350e1d89e guix-build-ce69e18947dd/output/aarch64-linux-gnu/SHA256SUMS.part
7a1b261af19acc88ae15899ddc5e31a0f555c79306d0fe91573af38dc02e15d5 guix-build-ce69e18947dd/output/aarch64-linux-gnu/bitcoin-ce69e18947dd-aarch64-linux-gnu-debug.tar.gz
ba3b7a12e413b50f081f098b0497aa469148c1c7d4e22b35f2f0b2a0bc8bd7d6 guix-build-ce69e18947dd/output/aarch64-linux-gnu/bitcoin-ce69e18947dd-aarch64-linux-gnu.tar.gz
cb5f014976f968aa31fe3f805b79d9b84dd7a476c87eb99317a84c4aa8066511 guix-build-ce69e18947dd/output/arm-linux-gnueabihf/SHA256SUMS.part
d94b52a00230cddfa8f7368a3698cbd80b3eb304942570b9f80c2b191aa66e15 guix-build-ce69e18947dd/output/arm-linux-gnueabihf/bitcoin-ce69e18947dd-arm-linux-gnueabihf-debug.tar.gz
5188c4519d8663702b47a3aa93c1e91044f055102986b5182f024ce229996279 guix-build-ce69e18947dd/output/arm-linux-gnueabihf/bitcoin-ce69e18947dd-arm-linux-gnueabihf.tar.gz
0d89495aa5a44695d09f8f8a7e40d9c20844fd168773816aa616a985b7500f91 guix-build-ce69e18947dd/output/dist-archive/bitcoin-ce69e18947dd.tar.gz
73349aad8d2fd446dbea3358d6234d94ef462b0e4bdf7440f741d5eb624da500 guix-build-ce69e18947dd/output/powerpc64-linux-gnu/SHA256SUMS.part
99ed51cbc7f1ad256c833e04979f318061ea318f5a023a52395fd201b2cb7aa3 guix-build-ce69e18947dd/output/powerpc64-linux-gnu/bitcoin-ce69e18947dd-powerpc64-linux-gnu-debug.tar.gz
0c261020682f4899f32f5c5d13613aacc426c0fa899f460b83970554f5b0bba0 guix-build-ce69e18947dd/output/powerpc64-linux-gnu/bitcoin-ce69e18947dd-powerpc64-linux-gnu.tar.gz
53c2a325973cf8af9c3b912a041373ee20aeb878e5c1ec4b93913e432f2c8dbe guix-build-ce69e18947dd/output/powerpc64le-linux-gnu/SHA256SUMS.part
af9068ff4858d503f06a7ea10949c5b31e5d7574cc37be435e954112e13a36b6 guix-build-ce69e18947dd/output/powerpc64le-linux-gnu/bitcoin-ce69e18947dd-powerpc64le-linux-gnu-debug.tar.gz
26237aaae06eff2e9553d7ffb9c24a0b2c1a0ca3362cfb9d56e4b11f36ce4676 guix-build-ce69e18947dd/output/powerpc64le-linux-gnu/bitcoin-ce69e18947dd-powerpc64le-linux-gnu.tar.gz
592f4f7657bb1a6bc405a7037220a7ff7e4eae2916669e7ace92134270f90861 guix-build-ce69e18947dd/output/riscv64-linux-gnu/SHA256SUMS.part
55020ffe7cea0eb37ed33f1c4335ee1c8c41d2b7bf8359d51389bd3c7d9fa973 guix-build-ce69e18947dd/output/riscv64-linux-gnu/bitcoin-ce69e18947dd-riscv64-linux-gnu-debug.tar.gz
f38cc1ba590aa4d3cbe82895ba139973636f6f28d2afc6b722d9517574f58288 guix-build-ce69e18947dd/output/riscv64-linux-gnu/bitcoin-ce69e18947dd-riscv64-linux-gnu.tar.gz
538c0e350646c60b47f1c43637af79e03269072db674a3327d566911ca4f45f7 guix-build-ce69e18947dd/output/x86_64-apple-darwin19/SHA256SUMS.part
78bf21a68537dee5a132a9b29cc56416a2453a6ac691873b2d0f2b7234d037e3 guix-build-ce69e18947dd/output/x86_64-apple-darwin19/bitcoin-ce69e18947dd-osx-unsigned.dmg
74aa94cc5bd3c954b16de3c850df8e3480e4585e9d02261df1109b42c3de019e guix-build-ce69e18947dd/output/x86_64-apple-darwin19/bitcoin-ce69e18947dd-osx-unsigned.tar.gz
b8cc87e6dab76add2e968adf6fe63e4c0a0c2d2df30ce15ebdb1790e813d73b2 guix-build-ce69e18947dd/output/x86_64-apple-darwin19/bitcoin-ce69e18947dd-osx64.tar.gz
73c281fc5abfff5ef21d8984a51ee5de27415db4a5c854569682e043b3486f28 guix-build-ce69e18947dd/output/x86_64-linux-gnu/SHA256SUMS.part
9f03730108957dc4d96f4567900e794d02144c9505b2e72241c8b324d1a35395 guix-build-ce69e18947dd/output/x86_64-linux-gnu/bitcoin-ce69e18947dd-x86_64-linux-gnu-debug.tar.gz
85f947c45218fc0f0ce4bfbe9b3a4098ae0401071c64d59c609eee8b3874ad80 guix-build-ce69e18947dd/output/x86_64-linux-gnu/bitcoin-ce69e18947dd-x86_64-linux-gnu.tar.gz
99f3e38758192fda710dc47543bd9b8ebc2cbbac4e25cd46862e8a98f261789e guix-build-ce69e18947dd/output/x86_64-w64-mingw32/SHA256SUMS.part
8cbb0cf7a8e88823c2bd1f583845640ef0358e8677620613c493288897fcaa31 guix-build-ce69e18947dd/output/x86_64-w64-mingw32/bitcoin-ce69e18947dd-win-unsigned.tar.gz
ec01895105ed10fd6a3ed5c5d6fa153526c268221e8a4af3497cb2b7d00855e1 guix-build-ce69e18947dd/output/x86_64-w64-mingw32/bitcoin-ce69e18947dd-win64-debug.zip
db04bb97728c3f9347e2ebaf87c298cdd0f936070bcf5a62ffd13badefb9dfb1 guix-build-ce69e18947dd/output/x86_64-w64-mingw32/bitcoin-ce69e18947dd-win64-setup-unsigned.exe
41bf36f1031bbc48a0dbad6bc14e71c60038cb3d11dbb82d300cfeec81e2914d guix-build-ce69e18947dd/output/x86_64-w64-mingw32/bitcoin-ce69e18947dd-win64.zip |
|
Code Review ACK ce69e18 |
|
Code review ACK ce69e18 |
|
waves goodbye to pixie 😢 |
sidhujag
pushed a commit
to syscoin/syscoin
that referenced
this pull request
Oct 13, 2021
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This finishes the transition to using LIEF for the ELF symbol and security checks.
Note that there's currently a work around used for identifying RISCV binaries (just checking the interpreter). I've sent a PR upstream, lief-project/LIEF#562, and we should be able to drop that when using LIEF 0.12.0 and onwards.