Skip to content

Conversation

jnewbery
Copy link
Contributor

@jnewbery jnewbery commented Apr 29, 2019

Carries out some remaining tidy-ups remaining after PR 15141:

  • split ValidationState into TxValidationState and BlockValidationState (commit from ajtowns)
  • various minor code style tidy-ups to the ValidationState class
  • remove the useless ret parameter from ValidationState::Invalid()
  • remove the now unused first_invalid parameter from ProcessNewBlockHeaders()
  • remove the fMissingInputs parameter from AcceptToMemoryPool(), and deal with missing inputs the same way as other errors by using the TxValidationState object.

Tip for reviewers (thanks ryanofsky!): The first commit ("[validation] Add CValidationState subclasses" ) is huge and can be easier to start reviewing if you revert the rote, mechanical changes:

Substitute the commit hash of commit "[validation] Add CValidationState subclasses" for in the commands below.

git checkout <CommitHash>
git grep -l ValidationState | xargs sed -i 's/BlockValidationState\|TxValidationState/CValidationState/g'
git grep -l ValidationResult | xargs sed -i 's/BlockValidationResult\|TxValidationResult/ValidationInvalidReason/g'
git grep -l MaybePunish | xargs sed -i 's/MaybePunishNode\(ForBlock\|ForTx\)/MaybePunishNode/g'
git diff HEAD^

After that it's possible to easily see the mechanical changes with:

git log -p -n1 -U0 --word-diff-regex=. <CommitHash>

@jnewbery
Copy link
Contributor Author

jnewbery commented Apr 29, 2019

This is built on #15141. Only commit [validation] Add CValidationState subclasses onwards are for review in this PR.

#15141 is merged. This is ready for review.

Copy link
Contributor

@promag promag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK, looks like a good refactor and improvement to add ValidationState subclasses.

@JeremyRubin
Copy link
Contributor

Concept ACK.

I'd like to see Invalid and ilk return void instead of always false, and clean up the call sites correspondingly. This adds a lines here and there, but I think it improves the readability of the code to see a false literal returned explicitly.

I also somewhat think that ideally there should be a third state class which handles Corruption cases for 'system state' or something. This covers the notion that the issue is not the fault of the block or the transaction, it is an issue with our local state or implementation. Not sure it's worth the churn though, two is already a huge improvement.

@laanwj
Copy link
Member

laanwj commented Apr 30, 2019

Concept ACK

@ajtowns
Copy link
Contributor

ajtowns commented May 3, 2019

I also somewhat think that ideally there should be a third state class which handles Corruption cases for 'system state' or something. This covers the notion that the issue is not the fault of the block or the transaction,

The current split is more about "what was being tested" not "what was at fault" -- that's easy to deal with via types, because you know what's being tested at compile time; but you don't know what's going to have been at fault at compile time, so I think dealing with that via the type system (ie as a third state class) wouldn't work.

@jnewbery jnewbery force-pushed the 2019-04-pr15141-cleanups branch from edcbe6c to 4a51148 Compare May 4, 2019 14:40
@jnewbery jnewbery force-pushed the 2019-04-pr15141-cleanups branch 2 times, most recently from 2b794dc to 4b4d081 Compare May 4, 2019 15:53
@DrahtBot
Copy link
Contributor

DrahtBot commented May 4, 2019

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #17268 (Epoch Mempool by JeremyRubin)
  • #17037 (Testschains: Many regtests with different genesis and default datadir by jtimon)
  • #16981 (Improve runtime performance of --reindex by LarryRuane)
  • #16974 (Walk pindexBestHeader back to ChainActive().Tip() if it is invalid by TheBlueMatt)
  • #16658 (validation: Rename CheckInputs to CheckInputScripts by jnewbery)
  • #16333 (test: Set BIP34Height = 2 for regtest by MarcoFalke)
  • #13533 ([tests] Reduced number of validations in tx_validationcache_tests by lucash-dev)
  • #10443 (Add fee_est tool for debugging fee estimation code by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@jnewbery jnewbery force-pushed the 2019-04-pr15141-cleanups branch from 4b4d081 to 8d88c86 Compare May 8, 2019 17:32
Handle this failure in the same way as all other failures: call Invalid()
with the reasons for the failure.
@jnewbery jnewbery force-pushed the 2019-04-pr15141-cleanups branch from 92584fe to 3004d5a Compare October 29, 2019 19:46
@jnewbery
Copy link
Contributor Author

squashed fixup commits.

Copy link
Contributor

@ryanofsky ryanofsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review ACK 3004d5a. Just whitespace change and pure virtual destructor added since last review.

@amitiuttarwar
Copy link
Contributor

amitiuttarwar commented Oct 29, 2019

code review ACK 3004d5a. Also built & ran tests locally.

@fjahr
Copy link
Contributor

fjahr commented Oct 29, 2019

Code review ACK 3004d5a . Only nit style change and pure virtual destructor added since my last review.

laanwj added a commit that referenced this pull request Oct 30, 2019
3004d5a [validation] Remove fMissingInputs from AcceptToMemoryPool() (John Newbery)
c428622 [validation] Remove unused first_invalid parameter from ProcessNewBlockHeaders() (John Newbery)
7204c64 [validation] Remove useless ret parameter from Invalid() (John Newbery)
1a37de4 [validation] Remove error() calls from Invalid() calls (John Newbery)
067981e [validation] Tidy Up ValidationResult class (John Newbery)
a27a295 [validation] Add CValidationState subclasses (John Newbery)

Pull request description:

  Carries out some remaining tidy-ups remaining after PR 15141:

  - split ValidationState into TxValidationState and BlockValidationState (commit from ajtowns)
  - various minor code style tidy-ups to the ValidationState class
  - remove the useless `ret` parameter from `ValidationState::Invalid()`
  - remove the now unused `first_invalid` parameter from `ProcessNewBlockHeaders()`
  - remove the `fMissingInputs` parameter from `AcceptToMemoryPool()`, and deal with missing inputs the same way as other errors by using the `TxValidationState` object.

  Tip for reviewers (thanks ryanofsky!): The first commit ("[validation] Add CValidationState subclasses" ) is huge and can be easier to start reviewing if you revert the rote, mechanical changes:

  Substitute the commit hash of commit "[validation] Add CValidationState subclasses" for <CommitHash> in the commands below.

  ```sh
  git checkout <CommitHash>
  git grep -l ValidationState | xargs sed -i 's/BlockValidationState\|TxValidationState/CValidationState/g'
  git grep -l ValidationResult | xargs sed -i 's/BlockValidationResult\|TxValidationResult/ValidationInvalidReason/g'
  git grep -l MaybePunish | xargs sed -i 's/MaybePunishNode\(ForBlock\|ForTx\)/MaybePunishNode/g'
  git diff HEAD^
  ```

  After that it's possible to easily see the mechanical changes with:

  ```sh
  git log -p -n1 -U0 --word-diff-regex=. <CommitHash>
  ```

ACKs for top commit:
  laanwj:
    ACK 3004d5a
  amitiuttarwar:
    code review ACK 3004d5a. Also built & ran tests locally.
  fjahr:
    Code review ACK 3004d5a . Only nit style change and pure virtual destructor added since my last review.
  ryanofsky:
    Code review ACK 3004d5a. Just whitespace change and pure virtual destructor added since last review.

Tree-SHA512: 511de1fb380a18bec1944ea82b513b6192df632ee08bb16344a2df3c40811a88f3872f04df24bc93a41643c96c48f376a04551840fd804a961490d6c702c3d36
@laanwj laanwj merged commit 3004d5a into bitcoin:master Oct 30, 2019
@jnewbery jnewbery deleted the 2019-04-pr15141-cleanups branch October 30, 2019 14:39
@jonatack
Copy link
Member

There oughta be a rule about merging review club PRs right before the meeting :)

@laanwj
Copy link
Member

laanwj commented Oct 30, 2019

@jonatack whoops …
nah, reviewing doesn't necessarily stop after merging 😄

@mzumsande
Copy link
Contributor

Ok, will still mention my code-review ACK for 3004d5a that I was just about to write when this got merged :-)

@maflcko
Copy link
Member

maflcko commented Oct 30, 2019

nah, reviewing doesn't necessarily stop after merging

Agree. The only downside is that review comments (with the reviewer's name) can not be included in the merge commit.

@jonatack
Copy link
Member

Could possibly use a review club tag 🏷️

Copy link
Member

@jonatack jonatack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Built at 3004d5a, ran tests, running bitcoind tailing the log.

Code review progress:

  • [validation] Add CValidationState subclasses
  • [validation] Tidy Up ValidationResult class
  • [validation] Remove error() calls from Invalid() calls
  • [validation] Remove useless ret parameter from Invalid()
  • [validation] Remove unused first_invalid parameter from ProcessNewBlo…
  • [validation] Remove fMissingInputs from AcceptToMemoryPool()

Interesting reviewer tip in the PR description; result here (without the right colors).

@laanwj
Copy link
Member

laanwj commented Oct 30, 2019

Could possibly use a review club tag label

Done

@jonatack
Copy link
Member

Perhaps helpful context for future readers/reviewers, here are two comments motivating a27a295 (CValidationState subclasses) from discussion in #15141:

@jonatack
Copy link
Member

Could possibly use a review club tag label

Done

Thanks!

@practicalswift
Copy link
Contributor

Reviewers of this PR are encouraged to review PR #17624 ("net: Fix an uninitialized read in ProcessMessage(…, "tx", …) when receiving a transaction we already have") which fixes a quite serious bug introduced in this PR. Luckily it was caught before being part of any release :)

laanwj added a commit that referenced this pull request Nov 28, 2019
…", …) when receiving a transaction we already have

73b96c9 net: Fix uninitialized read in ProcessMessage(...) (practicalswift)

Pull request description:

  Fix an uninitialized read in `ProcessMessage(…, "tx", …)` when receiving a transaction we already have.

  The uninitialized value is read and used on [L2526 in the case of `AlreadyHave(inv) == true`](https://github.com/bitcoin/bitcoin/blob/d8a66626d63135fd245d5afc524b88b9a94d208b/src/net_processing.cpp#L2494-L2526).

  Proof of concept being run against a `bitcoind` built with MemorySanitizer (`-fsanitize=memory`):

  ```
  $ ./p2p-uninit-read-in-conditional-poc.py
  Usage: ./p2p-uninit-read-in-conditional-poc.py <dstaddr> <dstport> <net>
  $ bitcoind -regtest &
  $ ./p2p-uninit-read-in-conditional-poc.py 127.0.0.1 18444 regtest
  SUMMARY: MemorySanitizer: use-of-uninitialized-value
  [1]+  Exit 77                 bitcoind -regtest
  $
  ```

  Proof of concept being run against a `bitcoind` running under Valgrind (`valgrind --exit-on-first-error`):

  ```
  $ valgrind -q --exit-on-first-error=yes --error-exitcode=1 bitcoind -regtest &
  $ ./p2p-uninit-read-in-conditional-poc.py 127.0.0.1 18444 regtest
  ==27351== Conditional jump or move depends on uninitialised value(s)
  [1]+  Exit 1                  valgrind -q --exit-on-first-error=yes --error-exitcode=1 bitcoind -regtest
  $
  ```

  Proof of concept script:

  ```
  #!/usr/bin/env python3

  import sys

  from test_framework.mininode import NetworkThread
  from test_framework.mininode import P2PDataStore
  from test_framework.messages import CTransaction, CTxIn, CTxOut, msg_tx

  def send_duplicate_tx(dstaddr="127.0.0.1", dstport=18444, net="regtest"):
      network_thread = NetworkThread()
      network_thread.start()

      node = P2PDataStore()
      node.peer_connect(dstaddr=dstaddr, dstport=dstport, net=net)()
      node.wait_for_verack()

      tx = CTransaction()
      tx.vin.append(CTxIn())
      tx.vout.append(CTxOut())
      node.send_message(msg_tx(tx))
      node.send_message(msg_tx(tx))
      node.peer_disconnect()
      network_thread.close()

  if __name__ == "__main__":
      if len(sys.argv) != 4:
          print("Usage: {} <dstaddr> <dstport> <net>".format(sys.argv[0]))
          sys.exit(0)
      send_duplicate_tx(sys.argv[1], int(sys.argv[2]), sys.argv[3])
  ```

  Note that the transaction in the proof of concept is the simplest possible, but really any transaction can be used. It does not have to be a valid transaction.

  This bug was introduced in #15921 ("validation: Tidy up ValidationState interface") which was merged in to `master` 28 days ago.

  Luckily this bug was caught before being part of any Bitcoin Core release :)

ACKs for top commit:
  jnewbery:
    utACK 73b96c9
  laanwj:
    ACK 73b96c9, thanks for discovering and reporting this before it ended up in a release.

Tree-SHA512: 7ce6b8f260bcdd9b2ec4ff4b941a891bbef578acf4456df33b7a8d42b248237ec4949e65e2445b24851d1639b10681c701ad500b1c0b776ff050ef8c3812c795
jasonbcox added a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Jul 9, 2020
Summary:
This is in preparation for the next commit, which removes the useless
`ret` parameter from ValidationState::Invalid().

error() is simply a convenience wrapper that calls LogPrintf and returns
false. Call LogPrintf explicitly and substitute the error() call for a
false bool literal.

Partial backport of Core [[bitcoin/bitcoin#15921 | PR15921]] : bitcoin/bitcoin@1a37de4

Test Plan: `ninja check check-functional`

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D6876
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Jul 9, 2020
…ckHeaders()

Summary:
No callers use the returned value in first_invalid. Remove it from the
function signature and don't set it in the function.

Partial backport of Core [[bitcoin/bitcoin#15921 | PR15921]] : bitcoin/bitcoin@c428622

Depends on D6873 only to be lint-free.

Test Plan: `ninja check`

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D6874
jasonbcox added a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Jul 10, 2020
Summary:
ValidationState::Invalid() takes a parameter `ret` which is returned to
the caller. All call sites set this to false. Remove the `ret` parameter
and just return false always.

Partial backport of Core [[bitcoin/bitcoin#15921 | PR15921]] : bitcoin/bitcoin@7204c64

Test Plan: `ninja check check-functional`

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D6885
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Jul 13, 2020
Summary:
Handle this failure in the same way as all other failures: call Invalid()
with the reasons for the failure.

Partial backport of Core [[bitcoin/bitcoin#15921 | PR15921]] : bitcoin/bitcoin@3004d5a

Test Plan: `ninja check check-functional`

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D6923
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Jul 14, 2020
Summary:
3004d5a12d09d94bfc4dee2a8e8f2291996a4aaf [validation] Remove fMissingInputs from AcceptToMemoryPool() (John Newbery)
c428622a5bb1e37b2e6ab2c52791ac05d9271238 [validation] Remove unused first_invalid parameter from ProcessNewBlockHeaders() (John Newbery)
7204c6434b944f6ad51b3c895837729d3aa56eea [validation] Remove useless ret parameter from Invalid() (John Newbery)
1a37de4b3174d19a6d8691ae07e92b32fdfaef11 [validation] Remove error() calls from Invalid() calls (John Newbery)
067981e49246822421a7bcc720491427e1dba8a3 [validation] Tidy Up ValidationResult class (John Newbery)
a27a2957ed9afbe5a96caa5f0f4cbec730d27460 [validation] Add CValidationState subclasses (John Newbery)

Pull request description:

  Carries out some remaining tidy-ups remaining after PR 15141:

  - split ValidationState into TxValidationState and BlockValidationState (commit from ajtowns)
  - various minor code style tidy-ups to the ValidationState class
  - remove the useless `ret` parameter from `ValidationState::Invalid()`
  - remove the now unused `first_invalid` parameter from `ProcessNewBlockHeaders()`
  - remove the `fMissingInputs` parameter from `AcceptToMemoryPool()`, and deal with missing inputs the same way as other errors by using the `TxValidationState` object.

  Tip for reviewers (thanks ryanofsky!): The first commit ("[validation] Add CValidationState subclasses" ) is huge and can be easier to start reviewing if you revert the rote, mechanical changes:

  Substitute the commit hash of commit "[validation] Add CValidationState subclasses" for <CommitHash> in the commands below.

  ```sh
  git checkout <CommitHash>
  git grep -l ValidationState | xargs sed -i 's/BlockValidationState\|TxValidationState/CValidationState/g'
  git grep -l ValidationResult | xargs sed -i 's/BlockValidationResult\|TxValidationResult/ValidationInvalidReason/g'
  git grep -l MaybePunish | xargs sed -i 's/MaybePunishNode\(ForBlock\|ForTx\)/MaybePunishNode/g'
  git diff HEAD^
  ```

  After that it's possible to easily see the mechanical changes with:

  ```sh
  git log -p -n1 -U0 --word-diff-regex=. <CommitHash>
  ```

ACKs for top commit:
  laanwj:
    ACK 3004d5a12d09d94bfc4dee2a8e8f2291996a4aaf
  amitiuttarwar:
    code review ACK 3004d5a12d09d94bfc4dee2a8e8f2291996a4aaf. Also built & ran tests locally.
  fjahr:
    Code review ACK 3004d5a12d09d94bfc4dee2a8e8f2291996a4aaf . Only nit style change and pure virtual destructor added since my last review.
  ryanofsky:
    Code review ACK 3004d5a12d09d94bfc4dee2a8e8f2291996a4aaf. Just whitespace change and pure virtual destructor added since last review.

Tree-SHA512: 511de1fb380a18bec1944ea82b513b6192df632ee08bb16344a2df3c40811a88f3872f04df24bc93a41643c96c48f376a04551840fd804a961490d6c702c3d36

Backport of Core [[bitcoin/bitcoin#15921 | PR15921]] and [[bitcoin/bitcoin#17624 | PR17624]] (small fix to 15921)

Followup to [[bitcoin/bitcoin#15141 | PR15141]]

Some differences reviewers will encounter:
1. RejectCodes (such as REJECT_INVALID) do not appear in the original PR due to an out-of-order backport, but we currently still support them, so it was retained.
2. Some files (notably tests) contain refactors not present in the original PR. This is due to a few out-of-order backports but otherwise harmless.

Depends on D6923, D6929, D6930

Test Plan: `ninja check check-functional-extended`

Reviewers: #bitcoin_abc, deadalnix

Reviewed By: #bitcoin_abc, deadalnix

Subscribers: deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D6860
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Dec 16, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.