To build it:

./contrib/guix/guix-clean
env HOSTS='x86_64-linux-gnu' ./contrib/guix/guix-build

To test it:

 ./guix-build-34bebad831a9/distsrc-34bebad831a9-x86_64-linux-gnu/src/bitcoind

An example of how to import sv2-ffi in a generic c++ project can be found here

Concept NACK for Rust, until a secure bootstrap without trusted third party binaries is practical.

Concept NACK for Rust, until a secure bootstrap without trusted third party binaries is practical.

Isn't that already achieved with guix?

I think this work is interesting, that said it might need more thought on how to interface with Core.

AFAICT, the proposed PR just link the library in the build system and create a new template encoder in bitcoind. Though it doesn't introduce a new thread where the logic would co-run with other Core's current threads (ThreadSocketHandler, ThreadOpenConnection, ...). I initially followed this approach when I proposed AltNet with #18988. After gathering feedbacks from other contributors, a separated process sounds a more flexible and safer approach.

Recently, I've made some progress in this direction by building on top of the multiprocess project (see the hacky branch https://github.com/ariard/bitcoin/commits/2021-07-altnet-lightning). Process separation is promising, though going further I'm aiming to have AltNet pluggin modules written in Rust, while interfacing in Core over the Cap'n Proto newer interfaces. The current bottleneck is re-writing libmultiprocess in Rust, see bitcoin-core/libmultiprocess#56 for more information. It's more work than expected though I've a branch half-written doing that, hope to make it public soon(tm).

I think such modules or the one proposed in the current PR could live beyond the Core repository to avoid burdening the maintenance effort and the concerns about Rust chain of trust, while consuming the stable interfaces to ease deployment across the ecosystem.

Glad to see progress on the miner infrastructure, that's really needed!

Strong Concept ACK. At a high level, this is incredibly important work for a few reasons:

• having a good block template producer in Bitcoin Core is critical for moving Bitcoin's mining decentralization forward over the coming years, the importance of which cannot be understated,
• Bitcoin Core having more visibility/control into how and when block templates get produced gives Bitcoin Core a ton more flexibility going forward to do things like pre-construct next-next-block templates in advance and push them out quicker, gives Bitcoin Core a ton more control of when transactions get included in a block by enabling "push" support for template updates instead of only being polled, etc,
• the inefficiencies of getblocktemplate have been a real source of pain for pool operators for some time, I'm excited its going away.

As for multi-process, I don't think this is a good candidate for multi-process - block template building is incredibly latency-sensitive, plus needs full access to the mempool and block template construction. At most, I'd think we could think of doing the block template construction in Core and then handing the template off to a new process which does the StratumV2 protocol work, but (a) that adds a bunch more latency for no reason, and (b) you'd probably just use the StratumV2 protocol to communicate with that other binary, so why? In general, one reason to consider rust for the protocol side of this is precisely because you don't need to worry as much about it being in the same binary - the actual network logic itself isn't going to corrupt memory.

Some thoughts.

• I don't object in general to adding a Rust-based dependency, as long as we can treat it as such. I don't personally know Rust, and suspect that many contributors/reviewers aren't as comfortable with Rust, so I think my criterion for this is whether it's something that we can treat as external, existing, known-to-work, code that's maintained separately - like say libevent or LevelDB, and not something that would expect detailed review from this project. Evidence that this same code is used in other contexts/projects would certainly help. The fact that this would be code that's only accessible through a local, optional, interface makes the safety analysis a lot easier.
• I haven't followed the state of affairs with StratumV2. If it's expected that a significant part of the mining ecosystem will end up depending on this protocol, I agree it makes sense to include it in Bitcoin Core, and agree with @TheBlueMatt that direct integration is the lowest-latency solution.
• I believe Guix indeed solves the concern of having bootstrapping the Rust compiler, at least to the same extent that our C++ compilers for release builds are bootstrapped too. Right, @dongcarl?

So conditionally on getting some confirmation on my assumptions above, (somewhat hesitant) Concept ACK.

No, Guix does not solve the bootstrapping problem, because Guix itself has a bootstrapping problem. You can't get Guix going without running their trusted blobs.

No, Guix does not solve the bootstrapping problem, because Guix itself has a bootstrapping problem. You can't get Guix going without running their trusted blobs.

That's correct (for now), but the situation is not Rust-specific. Our release builds are created with Guix, so the C++ compiler used for releases is subject to the same concerns.

• I believe Guix indeed solves the concern of having bootstrapping the Rust compiler, at least to the same extent that our C++ compilers for release builds are bootstrapped too. Right, @dongcarl?

Yes I think this is exactly right. Building Rust in Guix is not any less bootstrappable than building anything else (C++) in Guix.

Some context: Guix gained the ability to compile Rust v1.19 from mrustc v0.8 in 2018 (blog post), and this bootstrap method has been actively maintained, and now compiles Rust v1.29 from mrustc v0.9 (newer versions of rust (say v1.n) simply build using the previous version (v1.(n-1)) until it reaches v1.29).

Concept ACK on implementing a Stratum v2 Template Provider in Core ignoring language.

On the C++/Rust question it is dependent on other contributors being comfortable with it. This was discussed in today's Core dev meeting and has come up repeatedly in the past e.g. #17090. I would guess a C++ PR would be an easier/quicker merge but there may be a Rust approach that can work.

That's correct (for now), but the situation is not Rust-specific. Our release builds are created with Guix, so the C++ compiler used for releases is subject to the same concerns.

We support users building their own, without using Guix's trust-required toolchain.

As for multi-process, I don't think this is a good candidate for multi-process - block template building is incredibly latency-sensitive, plus needs full access to the mempool and block template construction.

Ah I see the low-latency requirement, yes effectively IPC/serialization to access the mempool is likely going to be a worrying performance penalty.

That said, one advantage splitting nicely the block template/mempool subsystems behind interfaces like it has been done for GUI/wallet/node is the ability offered to the end-user to have either one fully-fledged binary or a collection of process in function what make sense as a deployment.

One motivation could be to have fallback mempools connected to your block template construction in case of a malicious partition targeting your main one.

Update build system following @dongcarl suggestions.

TP is enabled via a config flag: --enable-template-provider
For now when compile with guix TP is enabled.

Big concept ACK on supporting the use of StratumV2, but slight concept NACK on bundling it with the introduction of Rust, barring some additional context/rationale. It'd be nice to see a compelling reason why Rust should be used for this particular feature other than the standard potpourri of "Rust is safer/more ergonomic in $X_GENERAL_WAY." Edit: I do see that this change pulls in a stratum Rust library, so maybe that's why?

I have a number of questions about how Rust will dovetail with the existing codebase (and this PR might not be the appropriate place to discuss them), but some that come to mind in the context of this change are

• how will use of Rust work with existing locks and lock annotations?
  • E.g., are we going to lose the use of our deadlock detection tooling within Rust code?
• what's the real cost of maintenance/additional complexity in the build system?
• how much third-party code does this actually entail dragging in? AFAIU, concurrency, e.g., is not a simple thing in Rust and often entails the use of third-party libs like tokio.

If Rust is really a critical part of doing this feature, maybe we should look at how to offer the raw data necessary (in a performant way) to construct any number of block template formats outside of Core, by supplying data in a generic format over e.g. Unix sockets.

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #25076 (guix: native GCC 10 toolchain for Linux builds by fanquake)
• #25037 (build: Create noinst_LTLIBRARIES conditionally by hebasto)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

If Rust is really a critical part of doing this feature, maybe we should look at how to offer the raw data necessary (in a performant way) to construct any number of block template formats outside of Core, by supplying data in a generic format over e.g. Unix sockets.

Hi @jamesob:

• why rust: main reason to use rust it to not rewrite an already working set of libraries, the libraries have been written to be used both in core (for the implementation of the TP) and in rust (for the implementation of the other Sv2 roles)
• how will Rust work with existing locks: do not know I will check
• cost of maintenance/additional complexity in the build system: IMHO is pretty insignificant.
  Support TP in the build system add:
  1. few lines in the config file in order to check if there is the flag --enable-template-provider and if the system that will build core has at least rustc 1.51.
  2. An m4 function pulled from the gnu repo ax_compare_version.
  3. A make file that call rustc with the correct flag and build the rust library then add the builded library to LDFLAGS so that is linked with bitocoind (this is the most complex addition to the build system and are only 159 LOC that do a very simple task)
  4. It modify 2 line in the guix build and add rustc 1.51 in the manifest
• how much third-party: the rust code added in order to build the TP in core do not have any external dependency but std.
• doing it externally: this was my original idea but, as stated also in the above post by @TheBlueMatt:
  block template building is incredibly latency-sensitive, plus needs full access to the mempool and block template construction

No objection to rust code. After all it is not different than wallet or qt code, which only a subset of all devs can understand, write and review. I guess the only thing needed here is review(ers).

Just leaving a note that I'm currently working on an implementation of the template provider that integrates with the the rust sv2 project

🐙 This pull request conflicts with the target branch and needs rebase.

_{Want to unsubscribe from rebase notifications on this pull request? Just convert this pull request to a "draft".}

• Is it still relevant? ➡️ Please solve the conflicts to make it ready for review and to ensure the CI passes.
• Is it no longer relevant? ➡️ Please close.
• Did the author lose interest or time to work on this? ➡️ Please close it and mark it 'Up for grabs' with the label, so that it can be picked up in the future.

We are currently working on changes, should be ready for review soon.

There hasn't been much activity lately and the patch still needs rebase. What is the status here?

• Is it still relevant? ➡️ Please solve the conflicts to make it ready for review and to ensure the CI passes.
• Is it no longer relevant? ➡️ Please close.
• Did the author lose interest or time to work on this? ➡️ Please close it and mark it 'Up for grabs' with the label, so that it can be picked up in the future.

we are implementing it in pure c++ without the use of the rust ffi lib so the PR is outdated