-
Notifications
You must be signed in to change notification settings - Fork 37.7k
Remove the mempool's NotifyEntryAdded and NotifyEntryRemoved signals #17477
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove the mempool's NotifyEntryAdded and NotifyEntryRemoved signals #17477
Conversation
Request review from @TheBlueMatt and @morcos @TheBlueMatt - you introduced the vtxConflicted vector in BlockConnected in 461e49f with comment 'This change adds a parameter to BlockConnectedDisconnected which lists the transactions which were removed from mempool due to confliction as a result of this operation. While its somewhat of a shame to make block-validation-logic generate a list of mempool changes to be included in its generated callbacks, fixing this isnt too hard.' |
63611e7
to
906b478
Compare
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
Note that this can change the order of notifications for the wallet. If the node connects block A with conflicted txs a1 a2 and block B with conflicted txs b1 and b2 in one ActivateBestChainStep, the old ordering would be:
and the new ordering is:
I'm pretty sure that's not a problem. Take a look at how the wallet currently handles conflicted transactions in the Line 1084 in cd6cb97
All the |
Are you sure about that? Note that change that is unconfirmed and not in the mempool is not counted toward our balance, so this would open up race conditions where a |
I don't think that's true:
In both of those cases, the change output is unspendable because it conflicts with a transaction included in the block, but in the current code the wallet doesn't know about it until slightly later. So arguably there's a window condition in the existing code where the wallet might try to spend change that it thinks is in the mempool, but is actually conflicted with the latest block (although in practice this doesn't matter - it's always the case that there might be a better block that conflicts our transactions that we just don't know about yet). (If this PR was changing the way we notify the wallet about transactions removed from the mempool because they're included in a block, then I'd agree with you. We don't want to introduce a window where the wallet knows the transaction is no longer in the mempool but doesn't yet know that it's in a block.) |
What if the user set min_conf to 1000 and rbf to false for that tx that spends the change that is only available for a short time erroneously? With those settings, the user should expect to create a transaction that could never possibly fail, no? |
Maybe I don't fully understand the scenario you're describing, but if the tx is spending change that has been conflicted out of the mempool but the wallet has not yet been notified then that transaction will definitely fail, since it's spending an output that doesn't exist in the main chain. Essentially, the difference for the wallet in this PR is that it'll get notified slightly earlier about transactions (and therefore possibly unconfirmed change outputs that it owns) that have been conflicted out of the mempool slightly before it is notified about the block that conflicts them. I can't think of any way that would cause windows of undesirable behaviour. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK.
I've tried to implement most of my comments here : https://github.com/ariard/bitcoin/commits/2019-11-review-17477, I think it's possible to remove the whole ConnectTrace
and replace by a simple typedef'ed vector.
What if the user set min_conf to 1000 and rbf to false for that tx that spends the change that is only available for a short time erroneously?
Your scenario is non-rbf tx B is spending change output from non-rbf confirmed tx A but tx B is finally conflicted and so your concern is about the window time between the conflict being connected in the validation logic and the wallet getting the info to act on it ? I think this change make it better as the window time is going to be shorted due to TransactionRemovedFromMempool triggered before BlockConnected.
src/validation.cpp
Outdated
* block are added via mempool callbacks prior to the BlockConnected() associated | ||
* with those transactions. If any transactions are marked conflicted, it is | ||
* assumed that an associated block will always be added. | ||
* | ||
* This class is single-use, once you call GetBlocksConnected() you have to throw | ||
* it away and make a new one. | ||
*/ | ||
class ConnectTrace { | ||
private: | ||
std::vector<PerBlockConnectTrace> blocksConnected; | ||
CTxMemPool &pool; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Commit 0f17fd9. I think after this one CTxMemPool member is no more of any use as it was used by ConnecTrace to connect to mempool signal, can you remove it ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
* | ||
* - EXPIRY (expired from mempool after -mempoolexpiry hours) | ||
* - SIZELIMIT (removed in size limiting if the mempool exceeds -maxmempool megabytes) | ||
* - REORG (removed during a reorg) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: "removed txn non-final anymore due to a reorg"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are several other reasons a tx could be removed for REORG:
- a spend from a coinbase output that is no longer mature (>100 confirmations)
- a descendants of non-final and non-mature outputs.
- if the re-org has been deep enough that the disconnect pool has filled up
- if the standardness or consensus rules have changed across the reorg
I'd rather leave this high level than try to enumerate all the possible reasons for this here.
* block are added via mempool callbacks prior to the BlockConnected() associated | ||
* with those transactions. If any transactions are marked conflicted, it is | ||
* assumed that an associated block will always be added. | ||
* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Commit 0f17fd9. Shoulnd't be more logic to remove both comments in previous commit (141f971) as it's the one where you effectively removed conflicted txn insertion and struct, Just looking at this commit atomically you can't make sense of "This class assumes (and asserts) that the conflicted transactions for a given block are added via mempool callbacks"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
@@ -70,6 +70,7 @@ | |||
#include <boost/algorithm/string/replace.hpp> | |||
#include <boost/algorithm/string/split.hpp> | |||
#include <boost/thread.hpp> | |||
#include <boost/signals2/signal.hpp> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
super-nit: signal before thread?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
@@ -698,9 +697,6 @@ class CTxMemPool | |||
|
|||
size_t DynamicMemoryUsage() const; | |||
|
|||
boost::signals2::signal<void (CTransactionRef)> NotifyEntryAdded; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it can be noted in commit message than NotifiyEntryAdded
signal didn't have any slot connected to and so was already useless
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
// one waiting for the transactions from the next block. We pop | ||
// the last entry here to make sure the list we return is sane. | ||
assert(!blocksConnected.back().pindex); | ||
blocksConnected.pop_back(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Commit 793b45c. I think this comment, assert and pop should also be removed in commit 141f971 as this is the last commit were their logic make sense. Also in BlockConnected
, the removed assert and emplace_back belonged to same logic.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I prefer to leave this as a separate commit since I think it's easier to review removing the logic piece-by-piece. If other reviewers agree with you, then I'm happy to squash the commits.
* Used to track blocks whose transactions were applied to the UTXO state as a | ||
* part of a single ActivateBestChainStep call. | ||
* | ||
* This class is single-use, once you call GetBlocksConnected() you have to throw |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should you keep this comment ? The class is still single-use.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
restored
906b478
to
23c2276
Compare
@ariard - thanks for the review. I've taken all of your comment suggestions and some of the changes in your branch. I've gone back on changing the |
@ariard : I think it's possible to actually remove If you agree, I'll remove the last two commits from this PR, and then just remove |
23c2276
to
e028f34
Compare
e028f34
to
cb155c9
Compare
rebased on master |
cb155c9
to
a352662
Compare
I've removed the final two commits (which tidied up the |
@jnewbery - your approach is better than the one tried in my aforementioned branch, because you avoid passing back and forth a |
a352662
to
227680f
Compare
Force-pushed a fix for a typo in a commit message. No code change. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice cleanup, concept ACK.
Looks like the first commit is the most sensible. Can't CWallet::ResendWalletTransactions -> CWalletTx::SubmitMemoryPoolAndRelay
mess if called in between TransactionRemovedFromMempool
and BlockConnected
?
@@ -2468,35 +2468,21 @@ static int64_t nTimePostConnect = 0; | |||
struct PerBlockConnectTrace { | |||
CBlockIndex* pindex = nullptr; | |||
std::shared_ptr<const CBlock> pblock; | |||
std::shared_ptr<std::vector<CTransactionRef>> conflictedTxs; | |||
PerBlockConnectTrace() : conflictedTxs(std::make_shared<std::vector<CTransactionRef>>()) {} | |||
PerBlockConnectTrace() {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit, just drop the constructor?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I plan to remove PerBlockConnectTrace()
entirely in #17562.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, but if you happen to push again I still think you could remove this line.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can't CWallet::ResendWalletTransactions -> CWalletTx::SubmitMemoryPoolAndRelay mess if called in between TransactionRemovedFromMempool and BlockConnected?
I'm not sure I fully understand the scenario. Can you explain further?
In master With this change "lots of stuff" can happen while the wallet state (transactions state really) is updated. So I was checking what can happen and one is: Lines 1747 to 1748 in 0ee914b
So it looks like a transaction can be broadcast again if this runs between bitcoin/src/node/transaction.cpp Lines 28 to 38 in 0ee914b
So I think there's no issue in this case. |
I agree, and I think this is similar to Marco's concern earlier and my response: #17477 (comment). This PR allows the wallet to be used in a small window between being notified of conflicted transactions from the block and the block itself. However, I don't think it opens any new window conditions in the wallet that don't already exist. Imagine the following scenario in master branch:
This is exactly the same behaviour as you describe above, and can already happen in the existing code. The only difference is the mempool removal reason, which is not reported to the wallet. |
rebased |
Pushed a fix to #17477 (comment). Thanks for the re-review @ryanofsky . I intend to address your other comments soon. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code review ACK 9317555
The only CValidationInterface client that cares about transactions that are removed from the mempool because of CONFLICT is the wallet. Start using the TransactionRemovedFromMempool method to notify about conflicted transactions instead of using the vtxConflicted vector in BlockConnected.
The wallet now uses TransactionRemovedFromMempool to be notified about conflicted wallet, and no other clients use vtxConflicted.
Since we don't add a vtxConflicted vector to BlockConnected the conflictedTxs member of PerBlockConnectTrace is no longer used.
ConnectTrace used to subscribe to the mempool's NotifyEntryRemoved callback to be notified of transactions removed for conflict. Since PerBlockConnectTrace no longer tracks conflicted transactions, ConnectTrace no longer requires these notifications.
It's no longer used for anything.
NotifyEntryAdded never had any subscribers so can be removed. Since ConnectTrace no longer subscribes to NotifyEntryRemoved, there are now no subscribers. The CValidationInterface TransactionAddedToMempool and TransactionRemovedFromMempool methods can now provide this functionality. There's no need for a special notifications framework for the mempool.
d70515a
to
e57980b
Compare
@ryanofsky - done. I've also addressed your inline comment. Let me know what you think. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Re-ACK e57980b
Retested this time after rebase on current master @ 9cc7eba, built, ran tests, ran bitcoind. Only change since last review is additional documentation: git diff b75fada..e57980b
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code review ACK e57980b, no code changes since previous review, but helpful new code comments have been added and the PR description is now more clear about where the old code came from
…EntryRe… …moved signals
Adjusted for the upstream removal of conflicts in BlockConnected and BlockDisconnected (bitcoin/bitcoin#17477). Since Namecoin has more types of conflicts due to name operations (as well as conflicts on block disconnect due to unexpired names), these functions had different signatures for Namecoin. With the upstream removal of conflicts from them, we can revert those changes back to the upstream code as much as possible.
…EntryRe… …moved signals
Summary: > The only CValidationInterface client that cares about transactions that > are removed from the mempool because of CONFLICT is the wallet. > > Start using the TransactionRemovedFromMempool method to notify about > conflicted transactions instead of using the vtxConflicted vector in > BlockConnected. This is a backport of Core [[bitcoin/bitcoin#17477 | PR17477]] [1/6] bitcoin/bitcoin@1168394 Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Differential Revision: https://reviews.bitcoinabc.org/D8864
Summary: > The wallet now uses TransactionRemovedFromMempool to be notified about > conflicted wallet, and no other clients use vtxConflicted. This is a backport of Core [[bitcoin/bitcoin#17477 | PR17477]] [2/6] bitcoin/bitcoin@cdb8934 Depends on D8864 See D6653 for differences in validationinterface.cpp Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Differential Revision: https://reviews.bitcoinabc.org/D8865
Summary: > Since we don't add a vtxConflicted vector to BlockConnected the > conflictedTxs member of PerBlockConnectTrace is no longer used. This is a backport of Core [[bitcoin/bitcoin#17477 | PR17477]] [3/6] bitcoin/bitcoin@5613f98 Depends on D8865 Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Differential Revision: https://reviews.bitcoinabc.org/D8866
Summary: > ConnectTrace used to subscribe to the mempool's NotifyEntryRemoved > callback to be notified of transactions removed for conflict. Since > PerBlockConnectTrace no longer tracks conflicted transactions, > ConnectTrace no longer requires these notifications. This is a backport of Core [[bitcoin/bitcoin#17477 | PR17477]] [4 & 5/6] bitcoin/bitcoin@969b65f bitcoin/bitcoin@2dd561f Backport note: these 2 commits have been squashed because the first one left an unused private attribute 'pool', causing a warning / error for build-clang Depends on D8866 Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Subscribers: majcosta Differential Revision: https://reviews.bitcoinabc.org/D8867
Summary: > NotifyEntryAdded never had any subscribers so can be removed. > > Since ConnectTrace no longer subscribes to NotifyEntryRemoved, there are > now no subscribers. > > The CValidationInterface TransactionAddedToMempool and > TransactionRemovedFromMempool methods can now provide this > functionality. There's no need for a special notifications framework for > the mempool. This concludes backport of Core [[bitcoin/bitcoin#17477 | PR17477]] [6/6] bitcoin/bitcoin@e57980b Depends on D8867 Note: See D6451 for why `NotifyEntryRemoved` has to be removed twice. It was probably duplicated by mistake. Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Differential Revision: https://reviews.bitcoinabc.org/D8869
…fication fix f2734f0 [Tests] Fix policyestimator and validation_block tests (random-zebra) 3448bc8 wallet: Minimal fix to restore conflicted transaction notifications (Antoine Riard) ba9d84d feature_notifications.py mimic upstream is_wallet_compiled() function to make future back ports easier. (furszy) 37c9944 [mempool] Remove NotifyEntryAdded and NotifyEntryRemoved callbacks (John Newbery) 5cc619f [validation] Remove pool member from ConnectTrace (John Newbery) 1e453b7 [validation] Remove NotifyEntryRemoved callback from ConnectTrace (John Newbery) e774bfd [validation] Remove conflictedTxs from PerBlockConnectTrace (John Newbery) 389680a [validation interface] Remove vtxConflicted from BlockConnected (furszy) 19c9383 [wallet] Notify conflicted transactions in TransactionRemovedFromMempool (John Newbery) 139882d Fire TransactionRemovedFromMempool from mempool This commit fires TransactionRemovedFromMempool callbacks from the mempool and cleans up a bunch of code. (furszy) Pull request description: PR built on top of #2209, #2235 and #2240 (don't get scared by the amount of commits, most of them are coming from 2209. Will disappear as soon as that one gets merged). Based on the brief talk originated in #2209 (comment) . Solving the conflicted transactions external listeners notification. Adapting the following PRs: bitcoin#14384, bitcoin#17477 and bitcoin#18982. Without functional test support, for the time being, for the lack of RBF functionality. ACKs for top commit: Fuzzbawls: ACK f2734f0 random-zebra: re-utACK f2734f0 and merging... Tree-SHA512: 07d437e0d5e5d9798b16d982b6f58585d1f0dcd82251c5ac06f47e44233433831243d451ce43068e33c6002a64b76fa4d165167a2cbaeeedc28cde2019853565
These boost signals were added in #9371, before we had a
TransactionRemovedFromMempool
method in the validation interface. TheNotifyEntryAdded
callback was used by validation to build a vector of conflicted transactions when connecting a block, which the wallet was notified of in theBlockConnected
CValidationInterface callback.Now that we have a
TransactionRemovedFromMempool
callback, we can fire that signal directly from the mempool for conflicted transactions.Note that #9371 was implemented to ensure
-walletnotify
events were fired for these conflicted transaction. We inadvertently stopped sending these notifications in #16624 (Sep 2019 commit 7e89994). We should probably fix that, but in a different PR.