-
Notifications
You must be signed in to change notification settings - Fork 37.7k
build: prune Boost headers in depends #24742
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Concept ACK. |
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsNo conflicts as of last run. |
Concept ACK. Or is this very long or unwieldy? 33mb of headers still sounds insane to me. |
There doesn't seem to be a super straight forward approach to doing this, hence why I opened it up for some brainstorming. One alternative is instead of trying to copy what we need, we just prune what we know we definitely don't need. I played around with this, and got the headers down to around 70mb, which is still much better than 170. This approach is also simpler code-wise, and probably easier to maintain across different HOSTS. There is still the potential of Boost introducing more cross-module dependency going forward, but I think the risk of that being a problem is low. |
I definitely like the white-list approach is better. It gives an exact overview of what we need. And might, at some point, when whittled down enough, include in the repository. Agree the other way around is bound to be easier.
Yes, whatever you do, this is something new that needs to be maintained when the boost version in depends changes. |
0255ef5
to
b59af8b
Compare
8b06e01
to
b58baa4
Compare
Sorry for all the force push noise, this now (CI passing) seems to be in a working state for all HOSTS. Boost Process is a mess, and is the primary reason for dragging in a huge amount of extra crap that we certainly don't otherwise use; i.e filesystem, system, asio, fusion, winapi + more. I've opened a few more minor PR's upstream to remove some deprecated header usage from Boost modules. Will run a Guix build shortly. |
GUIX hashes, mine match @fanquake
|
Rebased for #25465. |
cp boost/version.hpp $($(package)_staging_prefix_dir)/include/boost && \ | ||
cp boost/visit_each.hpp $($(package)_staging_prefix_dir)/include/boost && \ | ||
cp boost/weak_ptr.hpp $($(package)_staging_prefix_dir)/include/boost && \ | ||
cp -r boost/winapi $($(package)_staging_prefix_dir)/include/boost |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How painful is it to update this when boost changes?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not too awful, and should mostly be removing headers (there are already a few we can drop in the next Boost version). Note that the removal of the winapi dir here is because boost ships it twice (one in boost/winapi and another copy in boost/detail/winapi). If we can (hopefully) get rid of external signer there are a large amount of headers we could drop from here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes look correct, but I'm also curios about maintainability...
It's really nice to have a list of exactly what headers we need, nice work.
I'm nervous about this too. Is it possible to group the copies somewhat? Like "these are required for external signer"? Or are they so intertwined that it doesn't make sense to try? I'm also a little nervous about the procedure. When updating to a new boost version, it's possible that we accidentally forget to ship a new header and it gets picked up from a different version on the local filesystem instead, no? I guess we'd notice that quickly enough, but I don't love the trial/error requirement for updates. |
I think it's a bit too much of a mess. I had considered splitting up our depends boost package into multiple, and having options to turn various bits on and off, but, the overlap in headers, and complication in handling that probably isn't worth it. As mentioned above, I'd much rather us remove these nice-to-have, but no-one really uses (not 0, but in terms of instances of bitcoind, the % of users is close enough to 0) features, which are a blocker for removing/integrating the bare-minium of Boost we need (with a signals2 replacement this isn't too far-fetched). Dependencies like Boost Process remain fairly poorly maintained, in a adhoc fasion, by a single person upstream (very similar to upnpc and libnatpmp), and I think it continues to makes less and less sense for these kinds of dependencies to be a part of this project. |
I use external signer
There is no evidence for such a claim, aside from an issue wasn't detected by windows users |
c08c0a7
to
3d9af19
Compare
Dropped one additional unused header. |
Something similar was initially suggested in bitcoin#24301.
Added a commit to make this based on Boost 1.82.0. |
Closing for now. |
cc8dff5 depends: Boost 1.80.0 (fanquake) Pull request description: Mostly misc bug fixes and improvements, to the continually decreasing parts of Boost that we actually use. See: https://www.boost.org/users/history/version_1_80_0.html. Includes some boring upstreamed changes, i.e boostorg/multi_index#57, boostorg/signals2#60 that aid bitcoin#24742. Getting Boost modules to drop their usage of deprecated (redirect) headers means we can prune them from our depends tree. Also a requirement for bitcoin#25696. Guix Build (x86_64): ```bash e793a5ac9372b8fce6e19916be840eee99735bccdedf44b9bca006fd9ac8c395 guix-build-cc8dff5f8ff8/output/aarch64-linux-gnu/SHA256SUMS.part 001e0382f2b05e12f0ec5eaf09e001e31313ee3ab5367b0ba135ea2d7b863bf6 guix-build-cc8dff5f8ff8/output/aarch64-linux-gnu/bitcoin-cc8dff5f8ff8-aarch64-linux-gnu-debug.tar.gz a50229534b41eebd6c44001b56eb2be35b50d5f3a3b161a8fa46d7558b79693e guix-build-cc8dff5f8ff8/output/aarch64-linux-gnu/bitcoin-cc8dff5f8ff8-aarch64-linux-gnu.tar.gz d54d16ae4d5ef2bb0d0dcebbd6d1ec4b0f0976063bf66222a320e68340f902d5 guix-build-cc8dff5f8ff8/output/arm-linux-gnueabihf/SHA256SUMS.part 04c464043af256ee05d565040df9bb438151d4e041aa46f800fc567eba111c2f guix-build-cc8dff5f8ff8/output/arm-linux-gnueabihf/bitcoin-cc8dff5f8ff8-arm-linux-gnueabihf-debug.tar.gz 097cd2f633bf6d3b8e922d81ce9f6dfe793589418e394abfa4e183861f5ad236 guix-build-cc8dff5f8ff8/output/arm-linux-gnueabihf/bitcoin-cc8dff5f8ff8-arm-linux-gnueabihf.tar.gz 30c8074725b0701b1a781685f50a23ba038297b2633599f40792157daadc13cb guix-build-cc8dff5f8ff8/output/arm64-apple-darwin/SHA256SUMS.part df6036e89c7e8746badcbcf805ec4f84070847833c562bc2185c15904b1b909a guix-build-cc8dff5f8ff8/output/arm64-apple-darwin/bitcoin-cc8dff5f8ff8-arm64-apple-darwin-unsigned.dmg 8421c957b2b83c4afc3bc7f65f93798755acc99ac5921bd89b367c22c5b064d8 guix-build-cc8dff5f8ff8/output/arm64-apple-darwin/bitcoin-cc8dff5f8ff8-arm64-apple-darwin-unsigned.tar.gz 076e8e185bc01dd43feb3fcbcec04eff5ca435b0fdf47c6f05008feec1cf11ac guix-build-cc8dff5f8ff8/output/arm64-apple-darwin/bitcoin-cc8dff5f8ff8-arm64-apple-darwin.tar.gz 03de7ffcf8d98c61c3a5b763e9bcc6310c9895950f49768da2eeba4871d8478a guix-build-cc8dff5f8ff8/output/dist-archive/bitcoin-cc8dff5f8ff8.tar.gz 80ffe674a5fa86cfcf3f47eb8103dbfbc97dd74f793cf785c1347469f19b281f guix-build-cc8dff5f8ff8/output/powerpc64-linux-gnu/SHA256SUMS.part 8ce2024c3fa65e6cd613ef526f90c5904675012b22eb39ccf94081d91e14340a guix-build-cc8dff5f8ff8/output/powerpc64-linux-gnu/bitcoin-cc8dff5f8ff8-powerpc64-linux-gnu-debug.tar.gz 89dc31c0306b0f1c79e13e5f705b73233695a802fd4946aac03364c45e7de984 guix-build-cc8dff5f8ff8/output/powerpc64-linux-gnu/bitcoin-cc8dff5f8ff8-powerpc64-linux-gnu.tar.gz 7ed1b82ba3d4c1a8ddc24c6f650aa6b2d1be08580b2ec32cfec0fd682828d797 guix-build-cc8dff5f8ff8/output/powerpc64le-linux-gnu/SHA256SUMS.part 01c7f706b236b342f4e0b5043c5788168806338cdaefd5541b04118665c7e40b guix-build-cc8dff5f8ff8/output/powerpc64le-linux-gnu/bitcoin-cc8dff5f8ff8-powerpc64le-linux-gnu-debug.tar.gz 01767af5b190d0be4a075a5d038644b7dd0d1fc5f0b69698acaca92bfef960d0 guix-build-cc8dff5f8ff8/output/powerpc64le-linux-gnu/bitcoin-cc8dff5f8ff8-powerpc64le-linux-gnu.tar.gz 414de662e0bb3df6a5c8fc1a3576c90a943998b802d97070e31bde434fc6a7f1 guix-build-cc8dff5f8ff8/output/riscv64-linux-gnu/SHA256SUMS.part db55ddf8051e6dc80180f3eee12a7370bed1140959c34fb5de8d88f0344b23e2 guix-build-cc8dff5f8ff8/output/riscv64-linux-gnu/bitcoin-cc8dff5f8ff8-riscv64-linux-gnu-debug.tar.gz bb19f15545d5a52e800ffb7f141915ab7083fb4fab9a80269f6714b28a294502 guix-build-cc8dff5f8ff8/output/riscv64-linux-gnu/bitcoin-cc8dff5f8ff8-riscv64-linux-gnu.tar.gz d9241782d6e596ae02500ffe062501f80b064357bece5d10f7fd4d218240c3a1 guix-build-cc8dff5f8ff8/output/x86_64-apple-darwin/SHA256SUMS.part 1c3258a573e849a8efe6fce535a4f8737fb3a076ebe74ee29ca1e84a9113d24f guix-build-cc8dff5f8ff8/output/x86_64-apple-darwin/bitcoin-cc8dff5f8ff8-x86_64-apple-darwin-unsigned.dmg 7e6d562dd636fcacae88bedca45b49a879901c0fc1309ea1812aba59bbbcb5d0 guix-build-cc8dff5f8ff8/output/x86_64-apple-darwin/bitcoin-cc8dff5f8ff8-x86_64-apple-darwin-unsigned.tar.gz c4bc9c27466504417a4bd581cc024b8c5a370a51c8349c1f067752c98c2c14bc guix-build-cc8dff5f8ff8/output/x86_64-apple-darwin/bitcoin-cc8dff5f8ff8-x86_64-apple-darwin.tar.gz 59aaf11181373efd2e281004ce968967a26fa95f90c5a25f44cc4c687e9ddb0a guix-build-cc8dff5f8ff8/output/x86_64-linux-gnu/SHA256SUMS.part fbcde4bfe21314104c7e6036e1f7f4b3ef0e41a7546fef5d1f2a6456b955778d guix-build-cc8dff5f8ff8/output/x86_64-linux-gnu/bitcoin-cc8dff5f8ff8-x86_64-linux-gnu-debug.tar.gz b1302146a0e96f7faa150d764aa0ca92b46e887a886532ee7fc2b2cc63f174c5 guix-build-cc8dff5f8ff8/output/x86_64-linux-gnu/bitcoin-cc8dff5f8ff8-x86_64-linux-gnu.tar.gz 319b52c1a62a9cdad2e3f1fac8dd22458be2a9c1e6a0d60b33cb27272d69e52a guix-build-cc8dff5f8ff8/output/x86_64-w64-mingw32/SHA256SUMS.part 11300b916588cb060ac06e074b94bc5da852ab36446df903045ad593dce5056b guix-build-cc8dff5f8ff8/output/x86_64-w64-mingw32/bitcoin-cc8dff5f8ff8-win64-debug.zip 045fcb6ca721bdefb7490b3452f28449cb2b0449721dbbb20c174be76f96e1a3 guix-build-cc8dff5f8ff8/output/x86_64-w64-mingw32/bitcoin-cc8dff5f8ff8-win64-setup-unsigned.exe 38c826537c8054a35103e5ab7ca4f97ca98551f23bcbadb0532f6ca3444e0731 guix-build-cc8dff5f8ff8/output/x86_64-w64-mingw32/bitcoin-cc8dff5f8ff8-win64-unsigned.tar.gz 7c5f75c5a0b9b98540c8c779a6fc6f5e98d7de792d3a218e4ad7a68fa4027385 guix-build-cc8dff5f8ff8/output/x86_64-w64-mingw32/bitcoin-cc8dff5f8ff8-win64.zip ``` Guix Build (arm64): ```bash d54d16ae4d5ef2bb0d0dcebbd6d1ec4b0f0976063bf66222a320e68340f902d5 guix-build-cc8dff5f8ff8/output/arm-linux-gnueabihf/SHA256SUMS.part 04c464043af256ee05d565040df9bb438151d4e041aa46f800fc567eba111c2f guix-build-cc8dff5f8ff8/output/arm-linux-gnueabihf/bitcoin-cc8dff5f8ff8-arm-linux-gnueabihf-debug.tar.gz 097cd2f633bf6d3b8e922d81ce9f6dfe793589418e394abfa4e183861f5ad236 guix-build-cc8dff5f8ff8/output/arm-linux-gnueabihf/bitcoin-cc8dff5f8ff8-arm-linux-gnueabihf.tar.gz 30c8074725b0701b1a781685f50a23ba038297b2633599f40792157daadc13cb guix-build-cc8dff5f8ff8/output/arm64-apple-darwin/SHA256SUMS.part df6036e89c7e8746badcbcf805ec4f84070847833c562bc2185c15904b1b909a guix-build-cc8dff5f8ff8/output/arm64-apple-darwin/bitcoin-cc8dff5f8ff8-arm64-apple-darwin-unsigned.dmg 8421c957b2b83c4afc3bc7f65f93798755acc99ac5921bd89b367c22c5b064d8 guix-build-cc8dff5f8ff8/output/arm64-apple-darwin/bitcoin-cc8dff5f8ff8-arm64-apple-darwin-unsigned.tar.gz 076e8e185bc01dd43feb3fcbcec04eff5ca435b0fdf47c6f05008feec1cf11ac guix-build-cc8dff5f8ff8/output/arm64-apple-darwin/bitcoin-cc8dff5f8ff8-arm64-apple-darwin.tar.gz 03de7ffcf8d98c61c3a5b763e9bcc6310c9895950f49768da2eeba4871d8478a guix-build-cc8dff5f8ff8/output/dist-archive/bitcoin-cc8dff5f8ff8.tar.gz 80ffe674a5fa86cfcf3f47eb8103dbfbc97dd74f793cf785c1347469f19b281f guix-build-cc8dff5f8ff8/output/powerpc64-linux-gnu/SHA256SUMS.part 8ce2024c3fa65e6cd613ef526f90c5904675012b22eb39ccf94081d91e14340a guix-build-cc8dff5f8ff8/output/powerpc64-linux-gnu/bitcoin-cc8dff5f8ff8-powerpc64-linux-gnu-debug.tar.gz 89dc31c0306b0f1c79e13e5f705b73233695a802fd4946aac03364c45e7de984 guix-build-cc8dff5f8ff8/output/powerpc64-linux-gnu/bitcoin-cc8dff5f8ff8-powerpc64-linux-gnu.tar.gz 7ed1b82ba3d4c1a8ddc24c6f650aa6b2d1be08580b2ec32cfec0fd682828d797 guix-build-cc8dff5f8ff8/output/powerpc64le-linux-gnu/SHA256SUMS.part 01c7f706b236b342f4e0b5043c5788168806338cdaefd5541b04118665c7e40b guix-build-cc8dff5f8ff8/output/powerpc64le-linux-gnu/bitcoin-cc8dff5f8ff8-powerpc64le-linux-gnu-debug.tar.gz 01767af5b190d0be4a075a5d038644b7dd0d1fc5f0b69698acaca92bfef960d0 guix-build-cc8dff5f8ff8/output/powerpc64le-linux-gnu/bitcoin-cc8dff5f8ff8-powerpc64le-linux-gnu.tar.gz 414de662e0bb3df6a5c8fc1a3576c90a943998b802d97070e31bde434fc6a7f1 guix-build-cc8dff5f8ff8/output/riscv64-linux-gnu/SHA256SUMS.part db55ddf8051e6dc80180f3eee12a7370bed1140959c34fb5de8d88f0344b23e2 guix-build-cc8dff5f8ff8/output/riscv64-linux-gnu/bitcoin-cc8dff5f8ff8-riscv64-linux-gnu-debug.tar.gz bb19f15545d5a52e800ffb7f141915ab7083fb4fab9a80269f6714b28a294502 guix-build-cc8dff5f8ff8/output/riscv64-linux-gnu/bitcoin-cc8dff5f8ff8-riscv64-linux-gnu.tar.gz d9241782d6e596ae02500ffe062501f80b064357bece5d10f7fd4d218240c3a1 guix-build-cc8dff5f8ff8/output/x86_64-apple-darwin/SHA256SUMS.part 1c3258a573e849a8efe6fce535a4f8737fb3a076ebe74ee29ca1e84a9113d24f guix-build-cc8dff5f8ff8/output/x86_64-apple-darwin/bitcoin-cc8dff5f8ff8-x86_64-apple-darwin-unsigned.dmg 7e6d562dd636fcacae88bedca45b49a879901c0fc1309ea1812aba59bbbcb5d0 guix-build-cc8dff5f8ff8/output/x86_64-apple-darwin/bitcoin-cc8dff5f8ff8-x86_64-apple-darwin-unsigned.tar.gz c4bc9c27466504417a4bd581cc024b8c5a370a51c8349c1f067752c98c2c14bc guix-build-cc8dff5f8ff8/output/x86_64-apple-darwin/bitcoin-cc8dff5f8ff8-x86_64-apple-darwin.tar.gz 59aaf11181373efd2e281004ce968967a26fa95f90c5a25f44cc4c687e9ddb0a guix-build-cc8dff5f8ff8/output/x86_64-linux-gnu/SHA256SUMS.part fbcde4bfe21314104c7e6036e1f7f4b3ef0e41a7546fef5d1f2a6456b955778d guix-build-cc8dff5f8ff8/output/x86_64-linux-gnu/bitcoin-cc8dff5f8ff8-x86_64-linux-gnu-debug.tar.gz b1302146a0e96f7faa150d764aa0ca92b46e887a886532ee7fc2b2cc63f174c5 guix-build-cc8dff5f8ff8/output/x86_64-linux-gnu/bitcoin-cc8dff5f8ff8-x86_64-linux-gnu.tar.gz 319b52c1a62a9cdad2e3f1fac8dd22458be2a9c1e6a0d60b33cb27272d69e52a guix-build-cc8dff5f8ff8/output/x86_64-w64-mingw32/SHA256SUMS.part 11300b916588cb060ac06e074b94bc5da852ab36446df903045ad593dce5056b guix-build-cc8dff5f8ff8/output/x86_64-w64-mingw32/bitcoin-cc8dff5f8ff8-win64-debug.zip 045fcb6ca721bdefb7490b3452f28449cb2b0449721dbbb20c174be76f96e1a3 guix-build-cc8dff5f8ff8/output/x86_64-w64-mingw32/bitcoin-cc8dff5f8ff8-win64-setup-unsigned.exe 38c826537c8054a35103e5ab7ca4f97ca98551f23bcbadb0532f6ca3444e0731 guix-build-cc8dff5f8ff8/output/x86_64-w64-mingw32/bitcoin-cc8dff5f8ff8-win64-unsigned.tar.gz 7c5f75c5a0b9b98540c8c779a6fc6f5e98d7de792d3a218e4ad7a68fa4027385 guix-build-cc8dff5f8ff8/output/x86_64-w64-mingw32/bitcoin-cc8dff5f8ff8-win64.zip ``` ACKs for top commit: gruve-p: ACK bitcoin@cc8dff5 hebasto: ACK cc8dff5 jarolrod: ACK cc8dff5 Tree-SHA512: d34a4c3e84efe9cade6de9d7ba5aafc0a5c6efce83e313552a248f2b1eb49dc032a50cf4f1c7eb4767e754a7b67f75a129be19e36a892776b16001c31f6725be
The Boost 1.81.0 tarball is ~
118
mb, and expands to much larger than that, however we end up with ~150
mb of headers copied into the/include/boost
dir in depends. This is a lot by itself, and even more when it's170
mb * 9 (HOSTS for a guix build).With the changes in this PR, we end up with ~
50
mb of Boost headers in depends, which with some creative patching/pruning, could be trimmed even further. i.e sometimes you end up pulling in an entire boost module, because of a single include in another header we use, but in code that we don't actually need. In other cases there are deprecated headers which are still being used, which could be removed if the modules we care about stopped using them. I will open some PRs upstream to try and improve that situation, ie: boostorg/multi_index#57.