-
Notifications
You must be signed in to change notification settings - Fork 37.7k
Multiprocess bitcoin #10102
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Multiprocess bitcoin #10102
Conversation
Oh. Nice. Conceptually I think this goes into the right direction, though, I'm not sure if this could end up being only a temporary in-between step that may end up being replaced. Though, I'm aware that capnp has an RPC layer. But this would introduce another API (RPC / ZMQ / REST and then capnp RPC). I'm not saying this is the wrong direction, but we should be careful about adding another API. Three questions:
|
Reason this is currently using capnp is not performance but convenience. Capnp provides a high level API that supports bidirectional, synchronous, and asynchronous calls out of the box and allows me to easily explore implementation choices in bitcoin-qt without having to worry about low level protocol details, write a lot of parameter packing/unpacking boilerplate, and implement things like long polling. Capnp could definitely be replaced by JSON-RPC, though, and I've gone out of my way to support this by not calling capnp functions or using capnp types or headers anywhere except the
It could, but I'm going out of my way right now specifically NOT to add yet another bitcoind public API that could add to the JSON-RPC/REST/ZMQ/-blocknotify/-walletnotify confusion. The IPC here doesn't happen over a TCP port or even a unix socket path but over an anonymous socketpair using an inherited file descriptor. (I haven't done a windows implementation yet but similar things are possible there). I'm trying to make the change completely internal for now and transparent to users. Bitcoin-qt should still be invoked the same way and behave the same way as before, starting its own node and wallet. It just will happen to do this internally now by forking a bitcoind executable rather than calling in-process functions. This change will not add any new command line or GUI options allowing bitcoin-qt to connect to bitcoinds other than the one it spawns internally. Adding these features and supporting new public APIs might be things we want to do in the future, but they would involve downsides and complications that I'm trying to avoid here.
It's not required here because this change doesn't expose any new socket or endpoint, but it could be supported. Capnp's security model is based on capabilities, so to add authentication, you would just define a factory function that takes credentials as parameters and returns a reference to an object exposing the appropriate functionality. |
I'm really uncomfortable with using capn proto, but fine enough for some example testing stuff! I'm a fan of this general approach (ignoring the use of capn proto) and I think we should have done something like it a long time ago. |
strong concept ACK, but if is feasible, would prefer usage of the existing RPC instead of capn'proto |
Concept ACK, nice.
Please, let's not turn this into a discussion of serialization and RPC frameworks. To be honest that's been one of the things that's putting me off of doing work like this. If you want to suggest what framework to use, please make a thorough investigation of what method would be best to use for our specific use case, and propose that, but let's not start throwing random "I'm not comfortable with X" comments. We already use google protocol buffers in the GUI for payment requests to in a way that would be the straightforward choice. I'm also happy you didn't choose some XML-based abomonation or ASN.1. But anyhow, not here. For this pull it's fine to use whatever RPC mechanism you're comfortable with.
I'm also perfectly fine with keeping the scope here to "communication between GUI and bitcoind". This is not the place for introducing another external interface. Might be an option at some point in the future, but for now process isolation is enough motivation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated and rebased bf5f8ed -> 0ca73bc (pr/ipc.0 -> pr/ipc.1)
There's a lot of new changes here. More functions and callbacks have been wrapped, and there's now support for asynchronous calls that don't block event threads in the client and server.
At this point it would be very helpful to have code review for the main commit (0ca73bc "Add barebones IPC framework to bitcoin-qt and bitcoind"), because all the real infrastructure is now in place, and the main thing left to do is wrap up more functions and callbacks in IPC calls so the GUI can be functional.
Updated and rebased 0ca73bc -> 5e28c2f (pr/ipc.1 -> pr/ipc.3) to avoid a conflict. Main addition is an expanded src/ipc/README.md file. Again it would be very helpful to have some code review for the main commit (5e28c2f "Add barebones IPC framework to bitcoin-qt and bitcoind"). Giving feedback on the README file would be an easy place to start. |
5e28c2f
to
dda3756
Compare
Updated 5e28c2f -> dda3756 (pr/ipc.3 -> pr/ipc.4) This implements two suggestions from @JeremyRubin:
|
@laanwj pointed out in IRC (https://botbot.me/freenode/bitcoin-core-dev/msg/83983170/) that this change could help make the GUI more responsive by preventing Qt event processing from getting blocked, which currently happens in the monolithic At the time in IRC, I didn't think this change could directly help gui responsiveness, because although it does move libbitcoin and LOCK calls out of the However, this doesn't have to be the case. The place where IPC calls currently block waiting for responses is the But the This would add more overhead and make the average IPC call a little slower. But it would avoid situations where an unexpectedly slow IPC call ties up the whole gui, so it might be worth doing anyway. |
@ryanofsky Yes, integrating the IPC event loop and Qt event loop would help responsiveness. |
@ryanofsky I'm not familiar with Qt or capnproto, but I don't understand what the move to a different process has to do with making things less blocking. Any changes in architecture that would result in less blocks should equally be possible within the same process. |
I don't understand the goal here. On itself, there seems little benefit in separating the GUI and the rest into separate processes if those two processes still depend on each other (this is different from separating the wallet from the node, for example, as there as security considerations there... but for that use case the easiest approach seems to just have a lightweight mode and running two instances). I think it would be awesome if bitcoin-qt could be started and stopped independently to control a bitcoind process in the background, but if that's not the intent, what is the purpose? |
Let's say there are 50 places where bitcoin-qt calls a libbitcoin function. That means there are 50 places to update if you want bitcoin-qt handle to events while the function calls are executing. WIth the IPC framework, there is only one place you have to update instead of 50 places (if you want to do this).
Ok, so you think the benefits are small, and I think they are more significant.
This is trivial once bitcoin-qt is controlling bitcoind across a socket. I'm just implementing the socket part first, without introducing new UI features for now. |
Ok, that's what I was missing. It wasn't clear to me that this was a just first step towards a more useful separation. |
This doesn't crash currently because the method doesn't access any object members, but this behavior is fragile and incompatible with bitcoin#10102.
Spawn node subprocess instead of running node code internally
Spawn wallet subprocess instead of running wallet code internally
Add .wallet/.gui suffixes to log files created by bitcoin-gui and bitcoin-wallet processes so they don't clash with bitcoin-node log file.
… and logging 8a553c9 wallet: Add TxStateString function for debugging and logging (Ryan Ofsky) Pull request description: I found this useful while debugging silent conflict between bitcoin#10102 and bitcoin#27469 recently ACKs for top commit: ishaanam: utACK 8a553c9 achow101: ACK 8a553c9 furszy: Code ACK 8a553c9 Tree-SHA512: 87965c66bcb59a21e7639878bb567e583a0e624735721ff7ad1104eed6bb9fba60607d0e3de7be3304232b3a55f48bab7039ea9c26b0e81963e59f9acd94f666
6acec6b multiprocess: Add type conversion code for UniValue types (Ryan Ofsky) 0cc74fc multiprocess: Add type conversion code for serializable types (Ryan Ofsky) 4aaee23 test: add ipc test to test multiprocess type conversion code (Ryan Ofsky) Pull request description: Add type conversion hooks to allow `UniValue` objects, and objects that have `CDataStream` `Serialize` and `Unserialize` methods to be used as arguments and return values in Cap'nProto interface methods. Also add unit test to verify the hooks are working and data can be round-tripped correctly. The non-test code in this PR was previously part of bitcoin#10102 and has been split off for easier review, but the test code is new. --- This PR is part of the [process separation project](bitcoin#28722). ACKs for top commit: achow101: ACK 6acec6b dergoegge: reACK 6acec6b Tree-SHA512: 5d2cbc5215d488b876d34420adf91205dabf09b736183dcc85aa86255e3804c2bac5bab6792dacd585ef99a1d92cf29c8afb3eb65e4d953abc7ffe41994340c6
… and logging 8a553c9 wallet: Add TxStateString function for debugging and logging (Ryan Ofsky) Pull request description: I found this useful while debugging silent conflict between bitcoin#10102 and bitcoin#27469 recently ACKs for top commit: ishaanam: utACK 8a553c9 achow101: ACK 8a553c9 furszy: Code ACK 8a553c9 Tree-SHA512: 87965c66bcb59a21e7639878bb567e583a0e624735721ff7ad1104eed6bb9fba60607d0e3de7be3304232b3a55f48bab7039ea9c26b0e81963e59f9acd94f666
3b70f7b doc: fix broken doc/design/multiprocess.md links after bitcoin#24352 (Ryan Ofsky) 6d43aad span: Make Span template deduction guides work in SFINAE context (Ryan Ofsky) 8062c3b util: Add ArgsManager SetConfigFilePath method (Ryan Ofsky) 441d00c interfaces: Rename CalculateBumpFees methods to be compatible with capn'proto (Ryan Ofsky) 156f49d interfaces: Change getUnspentOutput return type to avoid multiprocess segfault (Ryan Ofsky) 4978754 interfaces: Add schedulerMockForward method so mockscheduler RPC can work across processes (Ryan Ofsky) 924327e interfaces: Fix const virtual method that breaks multiprocess support (Ryan Ofsky) 82a379e streams: Add SpanReader ignore method (Russell Yanofsky) Pull request description: This is a collection of small changes to interfaces and code which were needed as part of multiprocess PR bitcoin#10102, but have been moved here to make that PR smaller. All of these changes are refactoring changes which do not affect behavior of current code --- This PR is part of the [process separation project](bitcoin#28722). ACKs for top commit: achow101: ACK 3b70f7b naumenkogs: ACK 3b70f7b maflcko: re-ACK 3b70f7b 🎆 Tree-SHA512: 2368772b887056ad8a9f84c299cfde76ba45943770e3b5353130580900afa9611302195b899ced7b6e303b11f053ff204cae7c28ff4e12c55562fcc81119ba4c
6acec6b multiprocess: Add type conversion code for UniValue types (Ryan Ofsky) 0cc74fc multiprocess: Add type conversion code for serializable types (Ryan Ofsky) 4aaee23 test: add ipc test to test multiprocess type conversion code (Ryan Ofsky) Pull request description: Add type conversion hooks to allow `UniValue` objects, and objects that have `CDataStream` `Serialize` and `Unserialize` methods to be used as arguments and return values in Cap'nProto interface methods. Also add unit test to verify the hooks are working and data can be round-tripped correctly. The non-test code in this PR was previously part of bitcoin#10102 and has been split off for easier review, but the test code is new. --- This PR is part of the [process separation project](bitcoin#28722). ACKs for top commit: achow101: ACK 6acec6b dergoegge: reACK 6acec6b Tree-SHA512: 5d2cbc5215d488b876d34420adf91205dabf09b736183dcc85aa86255e3804c2bac5bab6792dacd585ef99a1d92cf29c8afb3eb65e4d953abc7ffe41994340c6
This fixes an error reported by Antoine Poinsot <darosior@protonmail.com> in bitcoin-core/libmultiprocess#123 that does not happen in master, but does happen with bitcoin#10102 applied, where if Ctrl-C is pressed when `bitcoin-node` is started, it is handled by both `bitcoin-node` and `bitcoin-wallet` processes, causing the wallet to shutdown abruptly instead of waiting for the node and shutting down cleanly. This change fixes the problem by having the wallet process print to stdout when it receives a Ctrl-C signal but not otherwise react, letting the node shut everything down cleanly.
This fixes an error reported by Antoine Poinsot <darosior@protonmail.com> in bitcoin-core/libmultiprocess#123 that does not happen in master, but does happen with bitcoin#10102 applied, where if the child bitcoin-wallet process is killed (either by an external signal or by Ctrl-C as reported in the issue) the bitcoin-node process will not shutdown cleanly after that because chain client stop() calls will fail. This change fixes the problem by handling ipc::Exception errors thrown during the stop() calls, and it relies on the fixes to disconnect detection implemented in bitcoin-core/libmultiprocess#160 to work effectively.
This fixes an error reported by Antoine Poinsot <darosior@protonmail.com> in bitcoin-core/libmultiprocess#123 that does not happen in master, but does happen with bitcoin#10102 applied, where if Ctrl-C is pressed when `bitcoin-node` is started, it is handled by both `bitcoin-node` and `bitcoin-wallet` processes, causing the wallet to shutdown abruptly instead of waiting for the node and shutting down cleanly. This change fixes the problem by having the wallet process print to stdout when it receives a Ctrl-C signal but not otherwise react, letting the node shut everything down cleanly.
This fixes an error reported by Antoine Poinsot <darosior@protonmail.com> in bitcoin-core/libmultiprocess#123 that does not happen in master, but does happen with bitcoin#10102 applied, where if the child bitcoin-wallet process is killed (either by an external signal or by Ctrl-C as reported in the issue) the bitcoin-node process will not shutdown cleanly after that because chain client stop() calls will fail. This change fixes the problem by handling ipc::Exception errors thrown during the stop() calls, and it relies on the fixes to disconnect detection implemented in bitcoin-core/libmultiprocess#160 to work effectively.
This fixes an error reported by Antoine Poinsot <darosior@protonmail.com> in bitcoin-core/libmultiprocess#123 that does not happen in master, but does happen with bitcoin#10102 applied, where if Ctrl-C is pressed when `bitcoin-node` is started, it is handled by both `bitcoin-node` and `bitcoin-wallet` processes, causing the wallet to shutdown abruptly instead of waiting for the node and shutting down cleanly. This change fixes the problem by having the wallet process print to stdout when it receives a Ctrl-C signal but not otherwise react, letting the node shut everything down cleanly.
This fixes an error reported by Antoine Poinsot <darosior@protonmail.com> in bitcoin-core/libmultiprocess#123 that does not happen in master, but does happen with bitcoin#10102 applied, where if the child bitcoin-wallet process is killed (either by an external signal or by Ctrl-C as reported in the issue) the bitcoin-node process will not shutdown cleanly after that because chain client stop() calls will fail. This change fixes the problem by handling ipc::Exception errors thrown during the stop() calls, and it relies on the fixes to disconnect detection implemented in bitcoin-core/libmultiprocess#160 to work effectively.
This fixes an error reported by Antoine Poinsot <darosior@protonmail.com> in bitcoin-core/libmultiprocess#123 that does not happen in master, but does happen with bitcoin#10102 applied, where if the child bitcoin-wallet process is killed (either by an external signal or by Ctrl-C as reported in the issue) the bitcoin-node process will not shutdown cleanly after that because chain client stop() calls will fail. This change fixes the problem by handling ipc::Exception errors thrown during the stop() calls, and it relies on the fixes to disconnect detection implemented in bitcoin-core/libmultiprocess#160 to work effectively.
Rename WITH_MULTIPROCESS to ENABLE_IPC, because ENABLE_IPC is a more accurate name for the feature. It controls whether the src/ipc/ directory is built and whether IPC features like -ipcbind, -ipcconnect, and -ipcfd are available. It does NOT currently enable multiprocess features which are implemented in bitcoin#10102 building on top of the IPC features. It will also no longer (as of the next commit), control whether a find_package call is made so the "WITH_" prefix is also inappropriate. -BEGIN VERIFY SCRIPT- git grep -l WITH_MULTIPROCESS | xargs sed -i s/WITH_MULTIPROCESS/ENABLE_IPC/g -END VERIFY SCRIPT-
ce7d94a doc: add release note (Sjors Provoost) 71f29d4 doc: update build and dependencies docs for IPC (Sjors Provoost) 3cbf747 cmake: set ENABLE_IPC by default (Sjors Provoost) 32a90e1 ci: use bitcoin-node for one depends job (Sjors Provoost) b333cc1 ci: build one depends job without multiprocess (Sjors Provoost) 16bce9a build: depends makes libmultiprocess by default (Sjors Provoost) Pull request description: Have depends make libmultiprocess by default. This PR causes the following behavior changes: 1. **bitcoin-node and bitcoin-gui binaries are included in releases**, due to `ENABLE_IPC` option being switched on by default in depends builds 2. `ENABLE_IPC` is also switched on by default in non-depends builds (instructions updated, #33190 does this as a standalone PR) 3. Various changes to CI: switching on `ENABLE_IPC` on in most configurations and using `bitcoin-node` binary (`bitcoin -m`) for functional tests in two of them. 4. The `bitcoin-node` and `bitcoin-gui` are added to `Maintenance.cmake` (since they're now in the release) This PR doesn't need to do all of these things at once. However it's is simpler, avoids code churn (especially in CI), and probably less confusing to make all these changes in the same PR. Windows is not supported yet, so `ENABLE_IPC` is off by default for it. It can be enabled after #32387. The initial main use case for IPC is to enable experimental support for the Mining IPC interface. A working example of a Stratum v2 Template Provider client using this interface can be found here: Sjors#48. See #31756 for discussion of when this should happen. Supersedes #30975. ## Wait what, why? The [Stratum v2 spec](https://stratumprotocol.org/specification) has been around for a few years now, mostly stable but with [ongoing activity](https://github.com/stratum-mining/sv2-spec/commits/main/) to clarify and fix more subtle issues encountered by implementers. Most of the implementation is built in Rust in a project called the Stratum Reference Implementation ([SRI](https://github.com/stratum-mining/stratum)). [Braiins](https://demand.work) added Stratum v2 support to both their (custom) firmware and pool several years ago, though they have fallen behind on recent spec changes (update: it seems they've fixed that). Apparently [new hardware is underway](#31802 (comment)) that supports Stratum v2 without the need for custom firmware. [DMND pool](https://www.dmnd.work) is Stratum v2 native from the start and employs several of the SRI developers (they haven't fully launched though). The industry is rather secretive, but apparently [there is more underway](#31802 (comment)). What does Bitcoin Core have to do with this? Well, in Stratum v2 jargon we are the Template Provider. Or at least, the Template Provider role needs us to make block templates. Initially back in 2023 the plan was to have Bitcoin Core implement this role entirely, see #23049. It would speak the sv2 encrypted message protocol. In fact the spec was designed around this assumption, making sure to only use cryptographic primitives already in our codebase. I took over that effort in late 2023, but during 2024 it became quite clear there was [strong resistance](#29432 (review)) to the idea of including all this new code, opening another network ports, etc. At the same time there was the long running multiprocess / IPC project #10102, and the idea was born to apply that here: instead of including Stratum v2 specific stuff, we offer a general Mining interface via an IPC connection that can e.g. push out fresh block templates as fees rise above a threshold (something not possible and/or very inefficient with `getblocktemplate`). A client sidecar application then sits between the Stratum v2 world and our node. Currently there's only one such sidecar application, maintained by me, and reusing the same codebase from the integrated approach. An attempt has been made to connect to our interface from Rust bitcoin-core/libmultiprocess#174, which would pave the way for SRI include the Template Provider role. Plebhash below indicates he's also working on that: #31802 (comment). So with this new approach in mind, between mid 2024 until spring 2025, I introduced a new Mining interface (#30200 - #31785). At the same time Russ Ryanosky worked on more tight integration of [libmultiprocess](https://github.com/bitcoin-core/libmultiprocess), including making it a subtree in #31741. See [design/multiprocess.md](https://github.com/bitcoin/bitcoin/blob/master/doc/design/multiprocess.md). Meanwhile I've been maintaining a fork of Bitcoin Core that includes the Template Provider, in the original integrated approach (Sjors#68) as well as an IPC + sidecar variant (Sjors#48). I've been shipping [regular releases](https://github.com/Sjors/bitcoin/releases), mostly after bug fixes or major rebases. The SRI team has been testing both variants, though the "official" [instruction on their web page](https://stratumprotocol.org/developers) is to stick to integrated version. Bug reports on [my repo fork](https://github.com/Sjors/bitcoin/issues?q=is%3Aissue) as well as on the [SRI repo](https://github.com/stratum-mining/stratum/issues?q=is%3Aissue%20%20label%3A%22template%20provider%22) are evidence of actual testing happening. But as Pavlenex writes below: > one recurring feedback I kept getting regardless of the size/type of miner is that the need to run a forked version of Bitcoin Core remains a significant barrier to adoption This PR gets rids of that significant barrier. People can download a "pristine" version of Bitcoin Core and the only change is to start it with `bitcoin node -m -ipcconnect=unix` instead of the usual `bitcoind`. Once that's released, I can dramatically simplify my sidecar codebase (Sjors#48) by removing pretty much all Bitcoin Core code that it doesn't need. My plan is to then make that a separate repository, which should be much easier to contribute to. I can then also make (deterministically built) signed releases, while making it clear that sidecar code has nothing to do with Bitcoin Core. Perhaps later on SRI implements the same and I can stop maintaining that project. Conceptually the situation will be a lot clearer; - today: download forked version of `bitcoind` (or a forked version of `bitcoin-node`, plus `bitcoin-mine`), install SRI stuff - tomorrow: download Bitcoin Core v30, install `bitcoin-mine` and SRI - future: download Bitcoin Core v30 and SRI <details> <summary> Guix hashes: </summary> ``` find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum 6dbf29baecb1d1593087ef1306ae7c78aa160c8beb04dc016e02549ae2d6d90d guix-build-ce7d94a492e6/output/aarch64-linux-gnu/SHA256SUMS.part 4b465e5e8f9652c176aa57cfe5c289267c28c3a3c684034a9ce471b529b95275 guix-build-ce7d94a492e6/output/aarch64-linux-gnu/bitcoin-ce7d94a492e6-aarch64-linux-gnu-debug.tar.gz 85bc6fa008b83419d96443d9dcc212b46f0992387fd58fd2dda5da76536ee22c guix-build-ce7d94a492e6/output/aarch64-linux-gnu/bitcoin-ce7d94a492e6-aarch64-linux-gnu.tar.gz 5ed9ea52a8bd55361d2d9c01fbd1b25ec9970530c2776e6c1959424ba1689f52 guix-build-ce7d94a492e6/output/arm-linux-gnueabihf/SHA256SUMS.part 2e483011fac64462d3aa000b577c3c05c825506032d879e39612e096d7a6c65b guix-build-ce7d94a492e6/output/arm-linux-gnueabihf/bitcoin-ce7d94a492e6-arm-linux-gnueabihf-debug.tar.gz 7ff1e3ba54944a2be89dd7d68cb91dff6f8950de9d7b521e15dfb746965f81bd guix-build-ce7d94a492e6/output/arm-linux-gnueabihf/bitcoin-ce7d94a492e6-arm-linux-gnueabihf.tar.gz abdf89e701b21b8c1238a8cec46aeaa55e0c3a0b88ad718636e89cde9813ca08 guix-build-ce7d94a492e6/output/arm64-apple-darwin/SHA256SUMS.part fb55cff0296cd5474811fe5cedcf28603628729dd085eeefa007c72582459b33 guix-build-ce7d94a492e6/output/arm64-apple-darwin/bitcoin-ce7d94a492e6-arm64-apple-darwin-codesigning.tar.gz e9aa566b1e79c467d7987b7c68fa608db788e6ddf89c4d90e524cd47b4faaf86 guix-build-ce7d94a492e6/output/arm64-apple-darwin/bitcoin-ce7d94a492e6-arm64-apple-darwin-unsigned.tar.gz bb428fc62a1230a55f49fa3b5c7ba8d588e8fed491357f890d5a6724a38b14e9 guix-build-ce7d94a492e6/output/arm64-apple-darwin/bitcoin-ce7d94a492e6-arm64-apple-darwin-unsigned.zip 5ef4b75e94b2c8265fbc588bbb42467a84438af969fddac0ea61ced3e4113345 guix-build-ce7d94a492e6/output/dist-archive/bitcoin-ce7d94a492e6.tar.gz 4f55d56a108c8f312a502cd5dfdf0840b091861a6d502df31caf4636a203697a guix-build-ce7d94a492e6/output/powerpc64-linux-gnu/SHA256SUMS.part 66c5b1242c60e37098885a00e24efe19baee4afcd2e3d6407207523d8872f055 guix-build-ce7d94a492e6/output/powerpc64-linux-gnu/bitcoin-ce7d94a492e6-powerpc64-linux-gnu-debug.tar.gz d9dbbee7217544eda26e77158cd82caeaef2b40fb9fc7033323e7ffe64264109 guix-build-ce7d94a492e6/output/powerpc64-linux-gnu/bitcoin-ce7d94a492e6-powerpc64-linux-gnu.tar.gz d9b808cc5685c819abcebb4ace65f003ebc4bfedf3fca046b34de37994358782 guix-build-ce7d94a492e6/output/riscv64-linux-gnu/SHA256SUMS.part eeeea470b1cf76515bfae14c779a3ea356d89f719d1fef1a81e8f0d6b04ab747 guix-build-ce7d94a492e6/output/riscv64-linux-gnu/bitcoin-ce7d94a492e6-riscv64-linux-gnu-debug.tar.gz 9993da4eb51618b8bd25ec88cc576496720e5589315e9eba6f3ddab25f9c3e60 guix-build-ce7d94a492e6/output/riscv64-linux-gnu/bitcoin-ce7d94a492e6-riscv64-linux-gnu.tar.gz 1b5a676580e0e79598d182f6ebbb05fb8aee2381edc3c09c042cae2600f448ab guix-build-ce7d94a492e6/output/x86_64-apple-darwin/SHA256SUMS.part 9152122d95a34d5df75305c6883c87707e7b09033fffd08e264d703ed177ef12 guix-build-ce7d94a492e6/output/x86_64-apple-darwin/bitcoin-ce7d94a492e6-x86_64-apple-darwin-codesigning.tar.gz 2793f75730dbef6bdf12b5ed7135e22ed21178abff2926dee92843837d4ab544 guix-build-ce7d94a492e6/output/x86_64-apple-darwin/bitcoin-ce7d94a492e6-x86_64-apple-darwin-unsigned.tar.gz e89aafd7e4a330a41f470e8f0a91ea876fad7d19547b404600867413f1a8ccb7 guix-build-ce7d94a492e6/output/x86_64-apple-darwin/bitcoin-ce7d94a492e6-x86_64-apple-darwin-unsigned.zip 955b27f881927a86da3c566357ad8ca68dbe17e9652bde8c482a57ceacba92cb guix-build-ce7d94a492e6/output/x86_64-linux-gnu/SHA256SUMS.part fd012be97bdf5c75ac12ddef21526bfdb5e17ecc77cde9c34d832194b0dc3293 guix-build-ce7d94a492e6/output/x86_64-linux-gnu/bitcoin-ce7d94a492e6-x86_64-linux-gnu-debug.tar.gz 0ecf7f80e9049369760d0e27fe6c026391ab25eae0f42336bef43e51a2621726 guix-build-ce7d94a492e6/output/x86_64-linux-gnu/bitcoin-ce7d94a492e6-x86_64-linux-gnu.tar.gz 2e8085f5fecc246d841b0bf6f28ecd0684a6cee49252fc88c1019d7586c7b7a2 guix-build-ce7d94a492e6/output/x86_64-w64-mingw32/SHA256SUMS.part c60041e8137eda352557254c5f67fb83eeb97ecfec342ee528451bd44ee4523a guix-build-ce7d94a492e6/output/x86_64-w64-mingw32/bitcoin-ce7d94a492e6-win64-codesigning.tar.gz b1be6b2f4de1c69c2e0e4de6dd97a4891ae9eb50d89435ef47247b5a187915a9 guix-build-ce7d94a492e6/output/x86_64-w64-mingw32/bitcoin-ce7d94a492e6-win64-debug.zip bfe143f41a20c537145c7044aca889b28efe19072b0150042a3bd865983b3d7e guix-build-ce7d94a492e6/output/x86_64-w64-mingw32/bitcoin-ce7d94a492e6-win64-setup-unsigned.exe 94a906b83d84db7b25f7e3cfdce2a2030243f2ee6cc70b1fc088459f0b2f382d guix-build-ce7d94a492e6/output/x86_64-w64-mingw32/bitcoin-ce7d94a492e6-win64-unsigned.zip ``` </details> ACKs for top commit: ryanofsky: Code review ACK ce7d94a. This was just rebased to fix a conflict since last review. josibake: ACK ce7d94a achow101: ACK ce7d94a ismaelsadeeq: ACK ce7d94a and tested again on macOS by building via depends and source. janb84: ACK ce7d94a Tree-SHA512: f7ab72933854e9dfce5746cdf764944bc26eec815f97cd0aa6b54fa499c3cccb1b678861ef5c1c793de28153d46bbb6e4d1b9aa0652163b74262e2d55ec8b813
🐙 This pull request conflicts with the target branch and needs rebase. |
This is a draft PR because it is based on #29409. The non-base commits are:
1d1906bc494c
test: Increase feature_block.py and feature_taproot.py timeouts0ee2322c6aa3
test: Fix multiprocess test for unclean shutdown on kille4497a4a209c
util: Add util::Result workaround to be compatible with libmultiprocessc695158cecf9
multiprocess: Add capnp serialization code for bitcoin typesc1b5d53711c1
multiprocess: Add capnp wrapper for Wallet interfaceef2d84cd307b
multiprocess: Add capnp wrapper for Node interface8d67dc6b1ca3
multiprocess: Make bitcoin-gui spawn a bitcoin-node processddbe8c2dbcab
multiprocess: Make bitcoin-node spawn a bitcoin-wallet processd3af681a2d05
multiprocess: Add debug.log .wallet/.gui suffixesf22746f5ecda
doc: Multiprocess misc doc and comment updatesc1e070d76166
combine_logs: Handle multiprocess wallet log filesThis PR adds an
--enable-multiprocess
configure option which builds newbitcoin-node
,bitcoin-wallet
, andbitcoin-gui
executables with relevant functionality isolated into different processes. See doc/design/multiprocess.md for usage details and future plans.The change is implemented by adding a new
Init
interface that spawns new wallet and node subprocesses that can be controlled over a socketpair by callingNode
,Wallet
, andChainClient
methods. When running with split processes, you can see the IPC messages going back and forth in-debug=1
output. Followup PR's #19460 and #19461 add-ipcbind
and-ipcconnect
options that allow more flexibility in how processes are connected.The IPC protocol used is Cap'n Proto, but this could be swapped out for another protocol. Cap'n Proto types and libraries are only accessed in the src/ipc/capnp/ directory, and not in any public headers or other parts of bitcoin code.
Slides from a presentation describing this change are available on google drive. Demo code used in the presentation was from an older version this PR (tag ipc.21, commits).
This PR is part of the process separation project.