The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.
A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #30443 (Introduce waitFeesChanged() mining interface by Sjors)
• #30437 (multiprocess: add bitcoin-mine test program by ryanofsky)
• #30409 (Introduce waitTipChanged() mining interface, replace RPCNotifyBlockChange and drop CRPCSignals by Sjors)
• #30205 (test: add mocked Sock that can read/write custom data and/or CNetMessages by vasild)
• #30051 (crypto, refactor: add new KeyPair class by josibake)
• #29775 (Testnet4 including PoW difficulty adjustment fix by fjahr)
• #28241 (Silent payment index (for light wallets and consistency check) by Sjors)
• #28201 (Silent Payments: sending by josibake)
• #28122 (Silent Payments: Implement BIP352 by josibake)
• #27433 (getblocktemplate improvements for segwit and sigops by Sjors)
• #26812 (test: add end-to-end tests for CConnman and PeerManager by vasild)
• #26697 (logging: use bitset for categories by LarryRuane)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Can you clarify what the point of PRs like this is, given you have a number of them open? They aren't reviewable, or mergable (given they are all based on various PRs, which them selves are in various stages of draft/WIP), and the continual pushes keep taking up all the CI resources in the repo (I thought we'd adjusted the CI so that you can use your own repo?).

@fanquake there's ongoing discussion in #29432 as to whether the Template Provider should be part of Bitcoin Core or be its own "sidecar" application. And the latter case there's still discussion about how to do that, e.g. using ZMQ/RPC, p2p or IPC.

I built a proof of concept for the IPC sidecar approach in Sjors#48.

It took me a while to iron out the interfaces, but with that (mostly) out of the way, I expect to need far fewer pushes than in the last few days.

Although the PoC works, which was easier than I expected, at the moment the sidecar approach is not shippable. E.g. I can't make reproducible binaries for it. It'll be at least 8 months until we can potentially ship a multiprocess capable Bitcoin Core release.

The other suggested approaches (RPC,ZMQ) don't even have a proof-of-concept demo yet.

So still consider #29432 to be plan A.

Every PR I've opened is reviewable. They focus on different aspects of the Template Provider. Having them separate should aid in review, since someone interested in the Noise Protocol #29346 may know nothing about connman #30332.

All interface PR's should be mergeable. I needed to make sure they're complete, which I think they are at this point. So I plan to mark #30443 and #30440 as ready for review soon.

I thought we'd adjusted the CI so that you can use your own repo?

That was intended for contributors to be able to build on top of #29432. I'll also use my own CI for anything more experimental than the Template Provider.

If the discussion in #29432 eventually settles on the sidecar approach (I don't think it has yet), then it makes sense that I move this PR, #29346, #30315 and #30332 over to my own repo. The other PRs, e.g. #30443, would stay here in that scenario as well.

Every PR I've opened is reviewable.

Sure, but it's unlikely anyone is going to review something like this, that is itself based on 7 other PRs, 4 of which are also marked as draft. Even just due to the fact that any review is likely wasted, due to that same review already being done in any of the base PRs.

If the discussion in #29432 eventually settles on the sidecar approach (I don't think it has yet), then it makes sense that I move this PR, #29346, #30315 and #30332 over to my own repo.

I think it would be good to let the discussion settle first, before opening all these PRs, just to (potentially) have to close / move them all again later.

I opened #29346, #30315 and #30332 months before the renewed discussion started. My approach in #29432 is no different from that of #27854 which has had multiple concept ACK's from experienced developers for more than a year now. So I don't think it was premature to open these sub PRs.

Even just due to the fact that any review is likely wasted, due to that same review already being done in any of the base PRs.

That's always a risk when stacking PRs. So far it hasn't happened. The description in each PR points to the base and parent. This method of splitting larger projects into stacked pull requests is not unique to Sv2.

Whether the code here eventually ends up in Bitcoin Core or in another project, it's still useful to get review on it, especially since people use it, including on mainnet (at tiny scale). That includes the TemplateProvider class that this PR introduces.

I wrote:

E.g. I can't make reproducible binaries for it

Looks like that is possible, see #30483. I'll give that a try and see if people are able to figure it out.

I'm looking into moving #30332 and this PR to my fork so reduce the stack a bit. However I keep getting seemingly spurious MSAN / TSAN failures which I've never seen here. So I'll need to iron out some bugs in self-hosted CI first.

See e.g. https://cirrus-ci.com/task/5819158305177600 for Sjors#50 which is identical to #30332 (which passed CI).

Trying to reproduce in isolation in Sjors/#51.

Managed to fix the CI issue.

Moving this PR to Sjors#49 to reduce the PR stack here a bit and not overwhelm CI.