Mempool: Add tracking of ancestor packages #7594
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
585fc90 to
740a22d
Compare
This was referenced Feb 25, 2016
740a22d to
b97a7e8
Compare
|
Needed rebase. @sipa Regarding your comment about adding exact ancestor state checking to |
|
I tried adding complete ancestor-state checks to I do expect this to be substantially slower, but I assume that since we can now run on mainnet with |
|
Untested ACK |
|
In |
|
@dgenr8 I agree that refactoring that code would make that function clearer, but as the content of |
remove is no longer called non-recursively, so simplify the logic and eliminate an unnecessary parameter
The work limit served to prevent the descendant walking algorithm from doing too much work by marking the parent transaction as dirty. However to implement ancestor tracking, it's not possible to similarly mark those descendant transactions as dirty without having to calculate them to begin with. This commit removes the work limit altogether. With appropriate chain limits (-limitdescendantcount) the concern about doing too much work inside this function should be mitigated.
This implements caching of ancestor state to each mempool entry, similar to descendant tracking, but also including caching sigops-with-ancestors (as that metric will be helpful to future code that implements better transaction selection in CreatenewBlock).
026d7a3 to
ce019bf
Compare
|
Needed rebase. |
|
Ok. Looking closer I see there's no real performance hit. I thought otherwise at first glance. |
|
ACK. Tested by running with -checkmempool=400 for a few days. |
laanwj
added a commit
that referenced
this pull request
Mar 17, 2016
ce019bf Check all ancestor state in CTxMemPool::check() (Suhas Daftuar) e2eeb5d Add ancestor feerate index to mempool (Suhas Daftuar) 72abd2c Add ancestor tracking to mempool (Suhas Daftuar) 76a7632 Remove work limit in UpdateForDescendants() (Suhas Daftuar) 5de2baa Rename CTxMemPool::remove -> removeRecursive (Suhas Daftuar) 7659438 CTxMemPool::removeForBlock now uses RemoveStaged (Suhas Daftuar)
|
utACK ce019bf |
codablock
pushed a commit
to codablock/dash
that referenced
this pull request
Dec 19, 2017
ce019bf Check all ancestor state in CTxMemPool::check() (Suhas Daftuar) e2eeb5d Add ancestor feerate index to mempool (Suhas Daftuar) 72abd2c Add ancestor tracking to mempool (Suhas Daftuar) 76a7632 Remove work limit in UpdateForDescendants() (Suhas Daftuar) 5de2baa Rename CTxMemPool::remove -> removeRecursive (Suhas Daftuar) 7659438 CTxMemPool::removeForBlock now uses RemoveStaged (Suhas Daftuar)
21 tasks
furszy
added a commit
to PIVX-Project/PIVX
that referenced
this pull request
Jan 23, 2021
9b9c616 Fix missing zapwallettxes mode in wallet_hd.py functional test (furszy) d6d0ad9 [logs] fix zapwallettxes startup logs (John Newbery) 006c503 [wallet] fix zapwallettxes interaction with persistent mempool (John Newbery) c6d45c6 Adapting and connecting mempool_persist.py functional test to the test runner. (furszy) 4f26a4e Control mempool persistence using a command line parameter. (John Newbery) 5d949de [Qt] Do proper shutdown (Jonas Schnelli) e60da98 Allow shutdown during LoadMempool, dump only when necessary (Jonas Schnelli) c0a0e81 Moving TxMempoolInfo tx member to CTransactionRef (furszy) f0c2255 Add mempool.dat to doc/files.md (furszy) 8e52226 Add DumpMempool and LoadMempool (Pieter Wuille) 44c635d Add AcceptToMemoryPoolWithTime function (Pieter Wuille) 6bbc6a9 Add feedelta to TxMempoolInfo (Pieter Wuille) 9979f3d [mempool] move removed and conflicts transaction ref list to vector. (furszy) 4f672c2 Make removed and conflicted arguments optional to remove (Pieter Wuille) e51c4b8 Bypass removeRecursive in removeForReorg (Pieter Wuille) 54cf7c0 Get rid of CTxMempool::lookup() entirely (furszy) 35bc2a9 Finished the switch CTransaction storage in mempool to CTransactionRef and introduced the timeLastMempoolReq coming from btc#8080. (furszy) d10583b An adapted version of btc@b5599147533103efea896a1fc4ff51f2d3ad5808 (furszy) cb4fc6c An adapted version of btc@ed7068302c7490e8061cb3a558a0f83a465beeea (furszy) 9645775 Split up and optimize transaction and block inv queues (furszy) 68bc68f Don't do mempool lookups for "mempool" command without a filter (furszy) 7624823 mapNextTx: use pointer as key, simplify value (furszy) 191c62e Return mempool queries in dependency order (Pieter Wuille) 23c9f3e Eliminate TX trickle bypass, sort TX invs for privacy and priority. (furszy) 6ebfd17 tiny test fix for mempool_tests (Alex Morcos) 8c0016e Check all ancestor state in CTxMemPool::check() (furszy) 91c6096 Add ancestor feerate index to mempool (Suhas Daftuar) 64e84e2 Add ancestor tracking to mempool This implements caching of ancestor state to each mempool entry, similar to descendant tracking, but also including caching sigops-with-ancestors (as that metric will be helpful to future code that implements better transaction selection in CreatenewBlock). (furszy) 8325bb4 Fix mempool limiting for PrioritiseTransaction Redo the feerate index to be based on mining score, rather than fee. (furszy) 1fa40ac Remove work limit in UpdateForDescendants() The work limit served to prevent the descendant walking algorithm from doing too much work by marking the parent transaction as dirty. However to implement ancestor tracking, it's not possible to similarly mark those descendant transactions as dirty without having to calculate them to begin with. This commit removes the work limit altogether. With appropriate chain limits (-limitdescendantcount) the concern about doing too much work inside this function should be mitigated. (furszy) ba32375 Rename CTxMemPool::remove -> removeRecursive remove is no longer called non-recursively, so simplify the logic and eliminate an unnecessary parameter (furszy) c30fa16 CTxMemPool::removeForBlock now uses RemoveStaged (furszy) Pull request description: Ending up 2020 with a large PR :). Included a good number of performance and privacy improvements over the mempool and inv processing/sending areas + added mempool cache dump/load. Almost finishing with #1726 work, getting closer to 9725, and getting closer to be able to decouple the message processing thread <-> wallet and the wallet <-> GUI locks dependencies (which will be another long story.. but well, step by step). The final goal is clear, a much faster syncing process using pivx-qt (a good speed up for pivxd as well), smoother visual navigation when big wallets are syncing, less thread synchronization issues, among all of the improvements that will be coming with the backports adaptations. Adapted the following PRs to our sources: bitcoin#7062 —> only eb30666. bitcoin#7174 —> complete. bitcoin#7562 —> only c5d746a. bitcoin#7594 —> complete. bitcoin#7840 —> complete. bitcoin#7997 —> complete. bitcoin#8080 —> complete bitcoin#8126 —> except e9b4780 (we don't have the mapRelay) and c2a4724 (we don't have the relay expiration vector). bitcoin#8448 —> complete bitcoin#8515 —> complete bitcoin#9408 —> complete bitcoin#9966 —> complete bitcoin#10330 —> complete ACKs for top commit: random-zebra: ACK 9b9c616 Fuzzbawls: ACK 9b9c616 Tree-SHA512: 186bd09bbb19b55ec0d46dc27291663a0df2d60d8238c661a8c9b8cbf3677b6f8a92fa56bd7346a3cb5293712eeccc5d6542ee3c441d35a4f61e7d12e2ce489a
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements caching of count, size, (modified) fee, and sigops of each mempool entry with its unconfirmed ancestors.
This is in preparation for adding support to
CreateNewBlockfor transaction selection based on fees-with-ancestors, for which I'll open another pull request soon.Note that in order to keep accurate ancestor state,
CTxMemPool::removeForBlockhas to walk the descendants of each in-block transaction (so that it's ancestor state can be reflected for the ancestor which was confirmed). The performance hit appears to be relatively insignificant (at least on the mempools I was looking at from 10/1/15 - 10/10/15): it amounted to an average 0.3ms, from 10.9ms to 11.2ms, on my almost 2 year old hardware. Moreover, I think if this performance was ever an issue we could do a bigger re-engineering ofConnectBlockso that block relay isn't held up by updating the mempool state.I would like to also expose this ancestor information via RPC, but will hold off until after #7292, so I might do that in a future PR rather than in this one. Also, the unit tests in this PR will be improved somewhat after #7539.