Skip to content

Conversation

Sjors
Copy link
Member

@Sjors Sjors commented Dec 24, 2024

Introduce checkBlock() on the Mining interface for IPC, as well as a new checkblock RPC that's easier to use than getblocktemplate in proposal mode, and supports weak blocks.

Rationale

Verifying block templates (no PoW)

Stratum v2 allows miners to generate their own block template. Pools may wish (or need) to verify these templates. This typically involves comparing mempools, asking miners to providing missing transactions and then reconstructing the proposed block.1 This is not sufficient to ensure a proposed block is actually valid. In some schemes miners could take advantage of incomplete validation2.

The Stratum Reference Implementation (SRI), currently the only Stratum v2 implementation, collects all missing mempool transactions, but does not yet fully verify the block.3 There's currently no way for it to do so, short of re-implementing Bitcoin Core validation code.

(although SRI could use this PR, the Template Provider role doesn't need it, so this is not part of #31098)

Verifying weak blocks (reduced PoW)

Stratum v2 decentralises block template generation, but still hinge on a centralised entity to approve these templates.

There used to be a decentralised mining protocol P2Pool4 which relied on (what we would today call) weak blocks. In such a system all participants perform verification and there is no privileged pool operator (that can reject a template).

P2Pool shares form a "sharechain" with each share referencing the previous share's hash. Each share contains a standard Bitcoin block header, some P2Pool-specific data that is used to compute the generation transaction (total subsidy, payout script of this share, a nonce, the previous share's hash, and the current target for shares), and a Merkle branch linking that generation transaction to the block header's Merkle hash.

IIUC only the header and coinbase of these shares / weak blocks were distributed amongst all pool participants and verified. This was sufficient at the time because fees were negligible, so payouts could simply be distributed based on work.

However, if P2Pool were to be resurrected today5, it would need to account for fees in a similar manner as PPLNS2 above. In that case the share chain should include the full weak blocks6. The client software would need a way to verify those weak blocks.

A similar argument applies to Braidpool.

Enter checkblock and its IPC counterpart checkBlock()

Given a serialised block, it performs the same checks as submitblock, but without updating the block index, chainstate, etc. The proof-of-work check can be bypassed entirely, for the first use of verifying block templates. Alternatively a custom target can be provided for the use case of weak blocks.

When called with check_pow=false the checkblock RPC is (almost) the same as getblocktemplate in proposal mode, and indeed they both call checkBlock() on the Mining interface.

Implementation details:

TestBlockValidity() is moved to the ChainstateManager and refactored to no longer return BlockValidationState. This avoids the new checkBlock() method needing to pass BlockValidationState over IPC.

The refactor commit contains additional changes explained in its commit message.

Another commit drops support for the unused BlockValidationState encoding.

This PR contains a simple unit test and a more elaborate functional test.

Potential followups:

  • if the block contains "better" transactions, (optionally) add them to our mempool 5
  • keep track of non-standard transactions7
  • allow rollback (one or two blocks) for pools to verify stale work / uncles

Footnotes

  1. https://github.com/stratum-mining/sv2-spec/blob/main/06-Job-Declaration-Protocol.md

  2. https://delvingbitcoin.org/t/pplns-with-job-declaration/1099/45?u=sjors 2

  3. https://github.com/stratum-mining/stratum/blob/v1.1.0/roles/jd-server/src/lib/job_declarator/message_handler.rs#L196

  4. https://en.bitcoin.it/wiki/P2Pool

  5. this improves compact block relay once the real block comes in 2

  6. The share chain client software could use compact block encoding to reduce its orphan / uncle rate. To make that easier, we could provide a reconstructblock RPC (reconstructBlock IPC) that takes the header and short ids, fills in a CBlock from the mempool (and jail) and returns a list of missing items. The share chain client then goes and fetches missing items and calls the method again. It then passes the complete block to checkblock. This avoids the need for others to fully implement compact block relay. Note that even if we implemented p2p weak block relay, it would not be useful for share chain clients, because they need to relay additional pool-specific data.

  7. https://delvingbitcoin.org/t/second-look-at-weak-blocks/805

furszy and others added 7 commits September 7, 2024 12:15
The 'subtractfeefromamount' arg is only boolean for sendtoaddress().
Other commands should never provide it as a boolean.
Following changes were made:

1) Catch and signal error for duplicate string destinations.
2) Catch and signal error for invalid value type.
3) Catch and signal error for string destination not found in tx outputs.
4) Improved 'InterpretSubtractFeeFromOutputInstructions()' code organization.
5) Added test coverage for all possible error failures.

Also, fixed two PEP 8 warnings at the 'wallet_sendmany.py' file:
- PEP 8: E302 expected 2 blank lines, found 1 at the SendmanyTest class declaration.
- PEP 8: E303 too many blank lines (2) at skip_test_if_missing_module() and set_test_params()
…l page sizes)

This aims to complete our test framework BDB parser to reflect
our read-only BDB parser in the wallet codebase. This could be
useful both for making review of bitcoin#26606 easier and to also possibly
improve our functional tests for the BDB parser by comparing with
an alternative implementation.
This change allows the entire `depends/<host_prefix>` directory to be
relocatable.
1. Check that outbound nodes are treated
the same as whitelisted connections for
the purposes of getdata delays

2. Add test case that demonstrates
download retries are preferentially
given to outbound (preferred) connections
even when multiple announcements are
considered ready.
By setting DANGER_DOCKER_BUILD_CACHE_HOST_DIR, the task-specific
docker images built during the CI run can be cached. This allows,
for example, ephemeral CI runners to reuse the docker images (or
layers of it) from earlier runs, by persisting the image cache
before the ephemeral CI runner is shut down. The cache keyed by
`CONTAINER_NAME`.

As --cache-to doesn't remove old cache files, the existing cache
is removed after a successful `docker build` and the newly cached
image is moved to it's location to avoid the cache from growing
indefinitly with old, unused layers.

When --cache-from doesn't find the directory, the cached version is
a cache-miss, or the cache can't be imported for whatever other reason,
it warns and `docker build` continues by building the docker image.

This feature is opt-in. The documentation for the cache type=local
can be found https://docs.docker.com/build/cache/backends/local/

This replaces bitcoin#31377
@DrahtBot
Copy link
Contributor

DrahtBot commented Dec 24, 2024

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31564.

Reviews

See the guideline for information on the review process.

Type Reviewers
Concept ACK tdb3

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #31974 (Drop testnet3 by Sjors)
  • #31649 (consensus: Remove checkpoints (take 2) by marcofleon)
  • #31283 (Add waitNext() to BlockTemplate interface by Sjors)
  • #29415 (Broadcast own transactions only via short-lived Tor or I2P connections by vasild)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@Sjors
Copy link
Member Author

Sjors commented Dec 24, 2024

I wrote on IRC:

Does anyone know what UpdateUncommittedBlockStructures is good for in the submitblock RPC?

To which @sipa responded:

Sjors[m]: it looks like it adds the "witness nonce" if missing (the witness for the coinbase input is a reserved field for adding future commitments, but it must be present, so that function adds a 0)

This PR omits UpdateUncommittedBlockStructures. It would be nice to come up with a test that shows what it does and fails on the current code.

Copy link
Contributor

@mzumsande mzumsande left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The newly introduced CheckNewBlock() looks very similar to the existing TestBlockValidity() which was just removed from the mining interface in bfc4e02 - it just seems to be a bit more verbose and allows the multiplier for the PoW. Did you consider merging these functions, so that we don't have multiple functions in validation that basically do the same thing?

Copy link
Contributor

@tdb3 tdb3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK

Really liking the idea of facilitating these type of systems.

Instead of a multiplier I could also allow a custom target. This would give weak block systems more flexibility on how to derive their difficulty.

I'd be in favor of this. It's more flexible, could prevent interface churn, and if the user wishes to create the custom target from a simple multiplication they could do the multiplication on their end.

src/pow.cpp Outdated
@@ -161,3 +161,19 @@ bool CheckProofOfWorkImpl(uint256 hash, unsigned int nBits, const Consensus::Par

return true;
}

bool CheckWeakProofOfWork(uint256 hash, unsigned int nBits, unsigned int multiplier, const Consensus::Params& params)
Copy link
Contributor

@tdb3 tdb3 Dec 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to ignore before more concept acks arrive.
CheckWeakProofOfWork() seems really similar to CheckProofOfWorkImpl(). Maybe I'm missing something simple. What was the rationale for creating the new function vs adding an argument?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just that I didn't want to touch CheckProofOfWorkImpl.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also applies to the new DeriveTarget method. I added the motivation for not changing CheckProofOfWorkImpl to the commit message in: d3710b0

@Sjors
Copy link
Member Author

Sjors commented Dec 27, 2024

The newly introduced CheckNewBlock() looks very similar to the existing TestBlockValidity()

It looks like I ended up with a similar design after a long detour. I'll look into combining these.

Instead of a multiplier I could also allow a custom target

@instagibbs is there any reason you went for a multiplier?

@instagibbs
Copy link
Member

is there any reason you went for a multiplier?

nothing off the top of my head

@Sjors
Copy link
Member Author

Sjors commented Dec 30, 2024

I refactored checkblock to take a target argument rather than a multiplier. In order to make this easier to implement and test, I also introduced a gettarget RPC and DeriveTarget() helper.

I then dropped the original TestBlockValidity and renamed CheckNewBlock to it. The generateblock and getblocktemplate RPC calls, as well
as BlockAssembler::CreateNewBlock use the new method.

If people prefer I could rewrite the git history to directly refactor TestBlockValidity, though I suspect the current approach is easier to review.

I split the test changes out into #31581, so will mark this PR as draft for now. While working on that I noticed a rather big difference between the original TestBlockValidity and my incarnation of it: the former required holding cs_main, which I think was wrong.

@DrahtBot
Copy link
Contributor

🚧 At least one of the CI tasks failed.
Debug: https://github.com/bitcoin/bitcoin/runs/34990349558

Hints

Try to run the tests locally, according to the documentation. However, a CI failure may still
happen due to a number of reasons, for example:

  • Possibly due to a silent merge conflict (the changes in this pull request being
    incompatible with the current code in the target branch). If so, make sure to rebase on the latest
    commit of the target branch.

  • A sanitizer issue, which can only be found by compiling with the sanitizer and running the
    affected test.

  • An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

@Sjors Sjors force-pushed the 2024/12/check-block branch 3 times, most recently from c961e83 to 2985291 Compare December 30, 2024 18:18
@Sjors Sjors marked this pull request as draft December 30, 2024 18:18
@Sjors
Copy link
Member Author

Sjors commented Dec 30, 2024

I might make a separate PR to consider whether to drop AssertLockHeld(cs_main) from the original TestBlockValidity.

If that ends up rejected, then I'll keep the existing TestBlockValidity calls as they are, i.e. drop the last two commits from this PR and leave them as a followup.

Related: #31563


While working on that I noticed a rather big difference between the original TestBlockValidity and my incarnation of it: the former required holding cs_main, which I think was wrong.

Update: I was confused, CheckBlock needs to be called with a lock on cs_main, and there's no need to "drop AssertLockHeld(cs_main)". That should allow for some simplifications.

A bpftrace script that logs information from the
net:*_connection tracepoints.

I've tested this script with bpftrace version 0.14.1 and v0.20.2.
A v3 onion address with a `:` and a five digit port has a length of
68 chars. As noted in
bitcoin#25832 (comment)
peers e.g. added via hostname might have a longer CNode::m_addr_name.
These might be cut off in tracing tools.
fanquake and others added 12 commits February 4, 2025 09:57
faf8fc5 lint: Call lint_commit_msg from test_runner (MarcoFalke)
fa99728 lint: Move commit range printing to test_runner (MarcoFalke)
fa673cf lint: Call lint_scripted_diff from test_runner (MarcoFalke)

Pull request description:

  The lint `commit-script-check.sh` can not be called from the test_runner at all and must be called manually. Also, some checks require `COMMIT_RANGE` to be set.

  Fix all issues by moving two lint checks into the test_runner. Also, the proper commit range is passed to the checks by the test_runner, so that the user no longer has to do it.

ACKs for top commit:
  kevkevinpal:
    reACK [faf8fc5](bitcoin@faf8fc5)
  willcl-ark:
    tACK faf8fc5

Tree-SHA512: 78018adc618d997508c226c9eee0a4fada3899cdfd91587132ab1c0389aea69127bafc3a900e90e30aca2c6bae9dcd6e6188ef287e91413bc63ee66fb078b1af
7426afb [p2p] assign just 1 random announcer in AddChildrenToWorkSet (glozow)
4c1fa6b test fix: make peer who sends MSG_TX announcement non-wtxidrelay (glozow)
2da46b8 pass P2PTxInvStore init args to P2PInterface init (glozow)
e3bd51e [doc] how unique_parents can be empty (glozow)
32eb6dc [refactor] assign local variable for wtxid (glozow)
18820cc multi-announcer orphan handling test fixups (glozow)
c4cc61d [fuzz] GetCandidatePeers (glozow)
7704139 [refactor] make GetCandidatePeers take uint256 and in-out vector (glozow)
6e4d392 [refactor] rename to OrphanResolutionCandidate to MaybeAdd* (glozow)
57221ad [refactor] move parent inv-adding to OrphanResolutionCandidate (glozow)

Pull request description:

  Followup to bitcoin#31397.

  Addressing (in order):
  bitcoin#31397 (comment)
  bitcoin#31397 (comment)
  bitcoin#31397 (comment)
  bitcoin#31397 (comment)
  bitcoin#31397 (comment)
  bitcoin#31397 (comment)
  bitcoin#31397 (comment)
  bitcoin#31658 (review)
  bitcoin#31397 (comment)

ACKs for top commit:
  instagibbs:
    reACK 7426afb
  marcofleon:
    reACK 7426afb
  mzumsande:
    Code Review ACK 7426afb
  dergoegge:
    Code review ACK 7426afb

Tree-SHA512: bca8f576873fdaa20b758e1ee9708ce94e618ff14726864b29b50f0f9a4db58136a286d2b654af569b09433a028901fe6bcdda68dcbfea71e2d1271934725503
…hain file

d9c8aac depends, refactor: Avoid hardcoding `host_prefix` in toolchain file (Hennadii Stepanov)

Pull request description:

  This PR allows the entire `depends/<host_prefix>` directory to be relocatable.

  Only `libevent` package configuration files are non-relocatable for the version `2.1.12-stable` we use now. However, this issue has been fixed upstream in libevent/libevent@1f1593f and friends.

  Fixes bitcoin#31050.

ACKs for top commit:
  theuni:
    Neat. utACK d9c8aac
  ryanofsky:
    Code review ACK d9c8aac

Tree-SHA512: c4c340722e63fc1da36fba2b15f025a6e1d477da1991194d678f546f2f3ea9454e2f0ff054aae6ae6c332a0781a597c3ce63b9018b46b8c258033f0d19efbef3
…and waitforblockheight rpc's

7e0db87 test: added additional coverage to waitforblock and waitforblockheight rpc's (kevkevinpal)

Pull request description:

  Similar to bitcoin#31746

  This adds test coverage to the `waitforblock` and `waitforblockheight` rpc's by adding a test to assert we get an rpc error if we include a negative timeout

ACKs for top commit:
  Sjors:
    utACK 7e0db87
  Prabhat1308:
    ACK [7e0db87](bitcoin@7e0db87)
  brunoerg:
    code review ACK 7e0db87
  BrandonOdiwuor:
    Code Review ACK 7e0db87

Tree-SHA512: c3b1b3a525e91e0092b783d73d2401126e3b8792a394be00374258f20cf3d619498e6625d3aad5e5ced29509c5eac828ee03c4ee66ba405b91e89be7e8b07311
846a138 func test: Expand tx download preference tests (Greg Sanders)

Pull request description:

  1. Check that outbound nodes are treated the same as whitelisted connections for
  the purposes of `getdata` delays

  2. Add test case that demonstrates download retries are preferentially
  given to outbound (preferred) connections
  even when multiple announcements are
  considered ready.

  `NUM_INBOUND` is a magic number large enough that it should fail over 90% of the time
  if the underlying outbound->preferred->PriorityComputer logic was broken. Bumping this
  to 100 peers cost another 14 seconds locally for the sub-test, so I made it pretty small.

ACKs for top commit:
  i-am-yuvi:
    tACK 846a138 good catch
  maflcko:
    ACK 846a138 🍕
  marcofleon:
    lgtm ACK 846a138

Tree-SHA512: 337aa4dc33b5c0abeb4fe7e4cd5e389f7f53ae25dd991ba26615c16999872542391993020122fd255af4c7163f76c1d1feb2f2d6114f12a364c0360d4d52b8c3
723440c test framework, wallet: rename get_scriptPubKey method to get_output_script (Alfonso Roman Zubeldia)
fa0232a test: add validation for gettxout RPC response (Alfonso Roman Zubeldia)

Pull request description:

  Added a new test in `test/functional/rpc_blockchain.py` to validate the gettxout RPC response. This new test ensures all response elements are verified, including `bestblock`, `confirmations`, `value`, `coinbase`, and `scriptPubKey` details.

  Also renamed the method `get_scriptPubKey` from `test/functional/test_framework/wallet.py` to the modern name `get_output_script` as suggested by maflcko (bitcoin#30226 (comment))

ACKs for top commit:
  fjahr:
    reACK 723440c
  maflcko:
    lgtm ACK 723440c
  brunoerg:
    code review ACK 723440c

Tree-SHA512: 3384578909d2e7548cef302c5b8a9fed5b82dfc942892503ad4a05e73f5cceafad1eab3af9a27e54aef3db7631f8935298d6b882c70d2f02a9a75b8e3c209b6c
e3622a9 tracing: document that peer addrs can be >68 chars (0xb10c)
b19b526 tracing: log_p2p_connections.bt example (0xb10c)
caa5486 tracing: connection closed tracepoint (0xb10c)
b2ad6ed tracing: add misbehaving conn tracepoint (0xb10c)
68c1ef4 tracing: add inbound connection eviction tracepoint (0xb10c)
4d61d52 tracing: add outbound connection tracepoint (0xb10c)
85b2603 tracing: add inbound connection tracepoint (0xb10c)

Pull request description:

  This adds five new tracepoints with documentation and tests for network connections:

  - established connections with `net:inbound_connection` and `net:outbound_connection`
  - closed connections (both closed by us or by the peer) with `net:closed_connnection`
  - inbound connections that we choose to evict with `net:evicted_inbound_connection`
  - connections that are misbehaving and punished with `net:misbehaving_connection`

  I've been using these tracepoints for a few months now to monitor connection lifetimes, re-connection frequency by IP and netgroup, misbehavior, peer discouragement, and eviction and more. Together with the two existing P2P message tracepoints they allow for a good overview of local P2P network activity. Also sort-of addresses bitcoin#22006 (comment).

  I've been back and forth on which arguments to include. For example, `net:evicted_connection` could also include some of the eviction metrics like e.g. `last_block_time`, `min_ping_time`, ... but I've left them out for now. If wanted, this can be added here or in a follow-up. I've tried to minimize a potential performance impact by measuring executed instructions with `gdb` where possible (method described [here](bitcoin#23724 (comment))). I don't think a few hundred extra instructions are too crucial, as connection opens/closes aren't too frequent (compared to e.g. P2P messages).   Note: e.g. `CreateNodeFromAcceptedSocket()` usually executes between 80k and 90k instructions for each new inbound connection.

  | tracepoint                 | instructions                                           |
  |----------------------------|--------------------------------------------------------|
  | net:inbound_connection     | 390 ins                                                |
  | net:outbound_connection    | between 700 and 1000 ins                                     |
  | net:closed_connnection     | 473 ins                                                |
  | net:evicted_inbound_connection     | not measured; likely similar to net:closed_connnection |
  | net:misbehaving_connection | not measured                                           |

  Also added a bpftrace (tested with v0.14.1) `log_p2p_connections.bt` example script that produces output similar to:
  ```
  Attaching 6 probes...
  Logging opened, closed, misbehaving, and evicted P2P connections
  OUTBOUND conn to 127.0.0.1:15287: id=0, type=block-relay-only, network=0, total_out=1
  INBOUND conn from 127.0.0.1:45324: id=1, type=inbound, network=0, total_in=1
  MISBEHAVING conn id=1, message='getdata message size = 50001'
  CLOSED conn to 127.0.0.1:15287: id=0, type=block-relay-only, network=0, established=1231006505
  EVICTED conn to 127.0.0.1:45324: id=1, type=inbound, network=0, established=1612312312
  ```

ACKs for top commit:
  laanwj:
    re-ACK e3622a9
  vasild:
    ACK e3622a9
  sipa:
    utACK e3622a9

Tree-SHA512: 1032dcac6fe0ced981715606f82c2db47016407d3accb8f216c978f010da9bc20453e24a167dcc95287f4783b48562ffb90f645bf230990e3df1b9b9a6d4e5d0
…e checking if they have sent a message

3f4b104 test: make sure we are on sync with a peer before checking if they have sent a message (Sergi Delgado Segura)

Pull request description:

  p2p_orphan_handling checks whether a message has not been requested slightly too soon, making the check always succeed. This passes unnoticed since the expected result is for the message to not have been received, but it will make the test not catch a relevant change that should make it fail.

  An easy way to check this is the case is to modify one of the test cases to force a request within the expected time, and check how the request is not seen. After the change, the test would crash as expected:

  ```diff
  index 963d924..30ab5f2035 100755
  --- a/test/functional/p2p_orphan_handling.py
  +++ b/test/functional/p2p_orphan_handling.py
  @@ -186,9 +185,12 @@ class OrphanHandlingTest(BitcoinTestFramework):
           parent_inv = CInv(t=MSG_WTX, h=int(tx_parent_arrives["tx"].getwtxid(), 16))
           assert_equal(len(peer_spy.get_invs()), 0)
           peer_spy.assert_no_immediate_response(msg_getdata([parent_inv]))
  +        txid = 0xdeadbeef
  +        peer_spy.send_and_ping(msg_inv([CInv(t=MSG_WTX, h=txid)]))

           # Request would be scheduled with this delay because it is not a preferred relay peer.
           self.nodes[0].bumpmocktime(NONPREF_PEER_TX_DELAY)
  +        peer_spy.assert_never_requested(int(txid))
           peer_spy.assert_never_requested(int(tx_parent_arrives["txid"], 16))
           peer_spy.assert_never_requested(int(tx_parent_doesnt_arrive["txid"], 16))
           # Request would be scheduled with this delay because it is by txid.
  ```

  It is worth noting that this is not seen in the cases where the message is expected to be received, because in such cases `assert_never_requested` is always after a `wait_....` method, which is already waiting for the node to sync on their end.

ACKs for top commit:
  i-am-yuvi:
    ACK 3f4b104
  instagibbs:
    ACK 3f4b104
  glozow:
    ACK 3f4b104

Tree-SHA512: 321a6605d630bed2217b6374e999dbb84da14138263dd8adf65fe3a6cd7981a50c873beced9cf05cb6d747a912e91017c58e7d4323d25449c87d83095ff4cba9
1973a9e test: fixes p2p_ibd_txrelay wait time (Sergi Delgado Segura)

Pull request description:

  `p2p_ibd_txrelay` expects no GETDATA to have been received by a peer after announcing a transaction. The reason is that the node is doing IBD, so transaction requests are not replied to. However, the way this is checked is wrong, and the check will pass even if the node **was not** in IBD.

  This is due to the mocktime not being properly initialized, so the check is always performed earlier than it should, making it impossible for the request to be there.

  This can be checked by modifying the test so the peer **is not doing IBD**, and checking how the test succeeds on that assert (even though it fails later on, given the nature of the test):

  ```diff
  index 882f5b5..3a69ae5860 100755
  --- a/test/functional/p2p_ibd_txrelay.py
  +++ b/test/functional/p2p_ibd_txrelay.py
  @@ -34,7 +34,7 @@ NORMAL_FEE_FILTER = Decimal(100) / COIN

   class P2PIBDTxRelayTest(BitcoinTestFramework):
       def set_test_params(self):
  -        self.setup_clean_chain = True
  +        # self.setup_clean_chain = True
           self.num_nodes = 2
           self.extra_args = [
               ["-minrelaytxfee={}".format(NORMAL_FEE_FILTER)],
  @@ -43,9 +43,11 @@ class P2PIBDTxRelayTest(BitcoinTestFramework):

       def run_test(self):
           self.log.info("Check that nodes set minfilter to MAX_MONEY while still in IBD")
  -        for node in self.nodes:
  -            assert node.getblockchaininfo()['initialblockdownload']
  -            self.wait_until(lambda: all(peer['minfeefilter'] == MAX_FEE_FILTER for peer in node.getpeerinfo()))
  +        # for node in self.nodes:
  +        #     assert node.getblockchaininfo()['initialblockdownload']
  +        #     self.wait_until(lambda: all(peer['minfeefilter'] == MAX_FEE_FILTER for peer in node.getpeerinfo()))
  ```

ACKs for top commit:
  i-am-yuvi:
    ACK 1973a9e
  glozow:
    ACK 1973a9e

Tree-SHA512: c4b3afe9927c5480671ebf5c1f6ee5fc7e3aeefeb13c210fa83587a6c126e1a8e40ad8e46587537d0f4bf06a36bbf2310ca065d685d4d9286e5a446b8d5b2235
Update the Transifex slug to match the new resource created for the
upcoming 29.x branch.
…ation`

f1d7a6d ci: Remove no longer needed '-Wno-error=documentation' (Hennadii Stepanov)

Pull request description:

  Picked from bitcoin#31726.

ACKs for top commit:
  maflcko:
    lgtm ACK f1d7a6d

Tree-SHA512: 2c4b197da1a60662922341062f9c1fe43b0e35a50194f4757057a48d7538f2f68540ed56508193a5c6a81e3f7dfbca78b15e3c101aae08d8ccd1fc5df0535932
TestBlockValidity(state, chainman.GetParams(), chainman.ActiveChainstate(), block, chainman.m_blockman.LookupBlockIndex(block.hashPrevBlock), /*fCheckPOW=*/false, /*fCheckMerkleRoot=*/true);
return BIP22ValidationResult(state);
std::string reason;
bool res{miner.checkBlock(block, {.check_pow = false}, reason)};
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to self: to maintain existing behavior this should emit LogError(). That's not necessary for generateblock since it already throws.

Emitting LogError() in TestBlockValidity would be wrong as of this PR, because that function doesn't know if we're checking our own blocks (which would indicate a fatal problem) or someone else's.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it's not needed. This code is for the proposal mode, where we verify a block given to us. It's not our node error if this block is invalid.

When we generate a block template ourselves through getblocktemplate or createNewBlock(), it goes through BlockAssembler::CreateNewBlock() which by default runs TestBlockValidity at the end, and throws if this return an error. No need to log if we're already crashing with std::runtime_error.

Some sources might be generated, and while they likely do not contain
any translatable strings, this change generalizes the approach to
include generated sources in the translation process as well.
Steps to reproduce the diff:
```
$ gmake -C depends -j $(nproc) MULTIPROCESS=1
$ cmake --preset dev-mode --toolchain depends/$(./depends/config.guess)/toolchain.cmake
$ cmake --build build_dev_mode --target translate
```
…release step

2f27c91 qt: Update the `src/qt/locale/bitcoin_en.xlf` translation source file (Hennadii Stepanov)
864386a cmake: Ensure generated sources are up to date for `translate` target (Hennadii Stepanov)
2b51dd3 Update Transifex slug for 29.x (Hennadii Stepanov)

Pull request description:

  This PR follows our [Release Process](https://github.com/bitcoin/bitcoin/blob/864386a7444fb5cf16613956ce8bf335f51b67d5/doc/release-process.md).

  It is required to open Transifex translations for v29.0, as scheduled in bitcoin#31029.

  The previous similar PR: bitcoin#30548.

  **Notes for reviewers:**

  1. This is the first release process conducted after migrating the build system to CMake. This revealed a bug, which is fixed in the second commit

  2. To reproduce the diff in the third commit, follow these steps:
  ```
  gmake -C depends -j $(nproc) MULTIPROCESS=1
  cmake --preset dev-mode --toolchain depends/$(./depends/config.guess)/toolchain.cmake
  cmake --build build_dev_mode --target translate
  ```

ACKs for top commit:
  stickies-v:
    ACK 2f27c91

Tree-SHA512: 325ce2418f218b82cc3b0a6c727473963455680cdf6383a85768613ed9e485974b2e52bd5b2e7a7472ad8ebe40bccb2884764d7f9e83dc10a587cd7892e0028b
Sjors added 5 commits February 7, 2025 09:52
A later commit adds checkBlock() to the Mining interface. In order to
avoid passing BlockValidationState over IPC, this commit first
refactors TestBlockValidity to return a boolean instead, and pass failure
reasons via a string.

TestBlockValidity is moved into ChainstateManager which allows some
simplifications.

Comments are expanded.

The ContextualCheckBlockHeader check is moved to after CheckBlock,
which is more similar to normal validation where context-free checks
are done first.

Validation failure reasons are no longer printed through LogError(),
since it depends on the caller whether this implies an actual bug
in the node, or an externally sourced block that happens to be invalid.
When called from getblocktemplate, via BlockAssembler::CreateNewBlock(),
this method already throws an std::runtime_error if validation fails.

Additionally it moves the inconclusive-not-best-prevblk check from RPC
code to TestBlockValidity.
The Mining interface avoids using BlockValidationState.
And use it in miner_tests, getblocktemplate and generateblock.
@Sjors
Copy link
Member Author

Sjors commented Mar 4, 2025

I opened a fresh PR #31981 that takes only the first three commits from this one, dropping the new checkblock RPC as well as the reduced target verification.

Assuming most IPC / RPC consumers will use some sort of Bitcoin library to process the contents of blocks and templates, it should be easy enough for them to check the PoW.

And if I drop the target argument from checkblock it's no different from the getblocktemplate RPC in proposal mode. Which itself is easy enough to use, see e.g. pool2win/p2pool-v2@72291c0.

It's likely still more performant to use IPC instead of RPC, so the new PR focusses on that.

@Sjors Sjors closed this Mar 4, 2025
achow101 added a commit that referenced this pull request Jun 19, 2025
a18e572 test: more template verification tests (Sjors Provoost)
10c9088 test: move gbt proposal mode tests to new file (Sjors Provoost)
94959b8 Add checkBlock to Mining interface (Sjors Provoost)
6077157 ipc: drop BlockValidationState special handling (Sjors Provoost)
74690f4 validation: refactor TestBlockValidity (Sjors Provoost)

Pull request description:

  This PR adds the IPC equivalent of the `getblocktemplate` RPC in `proposal` mode.

  In order to do so it has `TestBlockValidity` return error reasons as a string instead of `BlockValidationState`. This avoids complexity in IPC code for handling the latter struct.

  The new Mining interface method is used in `miner_tests`.

  It's not used by the `getblocktemplate` and `generateblock` RPC calls, see #31981 (comment)

  The `inconclusive-not-best-prevblk` check is moved from RPC
  code to `TestBlockValidity`.

  Test coverage is increased by `mining_template_verification.py`.

  Superseedes #31564

  ## Background

  ### Verifying block templates (no PoW)

  Stratum v2 allows miners to generate their own block template. Pools may wish (or need) to verify these templates. This typically involves comparing mempools, asking miners to providing missing transactions and then reconstructing the proposed block.[^0] This is not sufficient to ensure a proposed block is actually valid. In some schemes miners could take advantage of incomplete validation[^1].

  The Stratum Reference Implementation (SRI), currently the only Stratum v2 implementation, collects all missing mempool transactions, but does not yet fully verify the block.[^2]. It could use the `getblocktemplate` RPC in `proposal` mode, but using IPC is more performant, as it avoids serialising up to 4 MB of transaction data as JSON.

  (although SRI could use this PR, the Template Provider role doesn't need it, so this is _not_ part of #31098)

  [^0]: https://github.com/stratum-mining/sv2-spec/blob/main/06-Job-Declaration-Protocol.md
  [^1]: https://delvingbitcoin.org/t/pplns-with-job-declaration/1099/45?u=sjors
  [^2]: https://github.com/stratum-mining/stratum/blob/v1.1.0/roles/jd-server/src/lib/job_declarator/message_handler.rs#L196

ACKs for top commit:
  davidgumberg:
    reACK a18e572
  achow101:
    ACK a18e572
  TheCharlatan:
    ACK a18e572
  ryanofsky:
    Code review ACK a18e572 just adding another NONFATAL_UNREACHABLE since last review

Tree-SHA512: 1a6c29f45a1666114f10f55aed155980b90104db27761c78aada4727ce3129e6ae7a522d90a56314bd767bd7944dfa46e85fb9f714370fc83e6a585be7b044f1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.