-
Notifications
You must be signed in to change notification settings - Fork 37.7k
refactor: have CCoins* data managed under CChainState #16443
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: have CCoins* data managed under CChainState #16443
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it make sense to split this into dumb refactoring and then hard-to-review changes, like the on in init.cpp?
An early commit (scripted-diff or not) could add ChainstateActive::CoinsTip
, a dumb alias to return pcoinsTip;
and replace the usages. The later commit could then make the other changes in this pull.
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
e56af43
to
ceca2c9
Compare
ceca2c9
to
cce18b8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
utACK ceca2c9. This looks good and is very straightforward. Feel free to ignore all my suggestions below! (They are not important and possibly not coherent.)
63a3e4e
to
d0e1e37
Compare
Thanks for the feedback, @ryanofsky. I think I've incorporated all of it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK d0e1e37 minus pcoinsTip
removal.
Also I checked the other access to CoinsViews members via CoinsTip like in src/rpc/blockchain.cpp
or src/rpc/rawtransaction.cpp
. I've found every to be annotated or with a lock before, but was wandering if we shouldn't rely on EXCLUSIVE_LOCK_REQUIRED
if possible and not AssertLockHeld
. What's the exact diff between them ?
size_t cache_size_bytes, | ||
bool in_memory, | ||
bool should_wipe, | ||
std::string leveldb_name = "chainstate"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmmm is there any risk than having a default leveldb_name cause database collisions for different CChainState at some point ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems like something that would be caught pretty easily in tests if the confusion ever did happen, and I'm not sure if repeating "chainstate"
in a few different places inline is any better. That said I'm happy to make this change if others agree it seems worthwhile.
|
Yes |
d0e1e37
to
c860b44
Compare
Thanks for the look, @ariard. Feedback pushed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
utACK c860b44. There have just been a lot of small changes since last review: rebase for bilingual_str, many new code comments and commit comments, more MakeUniques and c++ references replacing raw pointers, avoiding recursive cs_main lock in gettxoutsetinfo.
re: #16443 (comment)
Is there a consistent policy on which one to favour ?
It's good to use EXCLUSIVE_LOCK_REQUIRED wherever possible. AssertLockHeld is older and not used as much recently, but still useful for compilers other than clang and cases where clang annotations don't work (sometimes when there are private mutexes, locks in callbacks and virtual methods).
In commit 53c8d59 refactor: have CCoins* data managed under CChainState
Could you elaborate a bit more why it is save to flush the chainstate when shutdown is requested during the init sequence? As I read the code, previously we wouldn't flush and now we flush an empty db (potentially not having the best block hash set, because genesis is not loaded?). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK c860b44, code looks good overall. Running into segfaults when testing
Show signature and timestamp
Signature:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
ACK c860b44ce3, code looks good overall. Running into segfaults when testing
-----BEGIN PGP SIGNATURE-----
iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
pUi8YAwAnzGBHNruwZp6wvxhnfnew86jtl74PDidtnPwvGX9JiMOeY07UF70JOnO
OwTXN9IV7ZSAQklT0qik7QWAodG0h5TARUsmhnBfITrBTka81h+6JIN/ZTLq+abC
katI8VltU56K8HeXEcYTXuPWN3pjJAFlVphLPgvtxp5EwsFvykK6iebSKAKFfe5w
dCxSTuL/G/T+PAQNMXxAGS5MiheRFHu7I70ES9ddRuH4qGkVVutnjY9skajcX2+a
w9m1s/fFovci/bOaYCfQqfYlWXLHnOS+ewl9kGOPC3re6l3byY4+Lr19cmrDlNE+
qp3igws7FnoOJOjETY5D5sBvd+AikZcBk9uvVhWubPpJ5uLHQJactj/UigJtfGsJ
rKCpwX7oN30ZeSRf3vjCDUqO6HKKHih9dsPhQ8ikjswfqqTuMU0oPoNL+9PSaZi7
k9X0qcTpsC0M+Da+K2egehus507NFwpLLnMAWftLhzQoIxdt4/YHNEf0QT7P2L5F
5zZwImCS
=zVl4
-----END PGP SIGNATURE-----
Timestamp of file with hash ccd43c0d933c3fca4055f6f6006fddbebd4ec81c67d0810961187dc2dab9838d -
src/validation.cpp
Outdated
{ | ||
CBlockIndex *pindexDelete = m_chain.Tip(); | ||
CBlockIndex* pindexDelete = m_chain.Tip(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are you removing the mempool lock annotation? Aren't the lock annotations in the header correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah you're right - dumb mistake on my part.
src/validation.cpp
Outdated
@@ -2297,7 +2326,8 @@ class ConnectTrace { | |||
* | |||
* The block is added to connectTrace if connection succeeds. | |||
*/ | |||
bool CChainState::ConnectTip(CValidationState& state, const CChainParams& chainparams, CBlockIndex* pindexNew, const std::shared_ptr<const CBlock>& pblock, ConnectTrace& connectTrace, DisconnectedBlockTransactions &disconnectpool) | |||
bool CChainState::ConnectTip(CValidationState& state, const CChainParams& chainparams, CBlockIndex* pindexNew, const std::shared_ptr<const CBlock>& pblock, ConnectTrace& connectTrace, DisconnectedBlockTransactions& disconnectpool) | |||
EXCLUSIVE_LOCKS_REQUIRED(::cs_main) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same
I am seeing segmentation faults when shutting down during init:
|
c860b44
to
8cd527d
Compare
Good catches, @MarcoFalke! I've pushed fixes, including e4b3247 to solve the init bug you found. There are a few ways of addressing that one but I think an early exit from |
8cd527d
to
4f44bd2
Compare
I'm also curious about the answer to this. Would be good to say what's motivating the move in the comment. I guess I also had a similar question previously: #15606 (comment) |
4f44bd2
to
4e307e5
Compare
@ryanofsky @MarcoFalke good points. I've abandoned trying to reorder the init.cpp stuff and now have separated the construction of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
utACK 4e307e5. Only changes since last review are adding and using little InitCoinsViews / HasCoinsViews helpers so the init order doesn't have to change, and reverting the new FlushStateToDisk check which is no longer needed because the init order is not changing. Also, Marco's name is now added to the credits, but I can live with that.
-Thanks to Russell Yanofsky for helpful feedback.
+Thanks to Russell Yanofsky and Marco Falke for helpful feedback.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK 4e307e5 (reviewed changes since last one and ran tests)
@@ -314,7 +314,8 @@ bool CheckSequenceLocks(const CTxMemPool& pool, const CTransaction& tx, int flag | |||
// Returns the script flags which should be checked for a given block | |||
static unsigned int GetBlockScriptFlags(const CBlockIndex* pindex, const Consensus::Params& chainparams); | |||
|
|||
static void LimitMempoolSize(CTxMemPool& pool, size_t limit, unsigned long age) EXCLUSIVE_LOCKS_REQUIRED(pool.cs) | |||
static void LimitMempoolSize(CTxMemPool& pool, size_t limit, unsigned long age) | |||
EXCLUSIVE_LOCKS_REQUIRED(pool.cs, ::cs_main) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Based on above discussion on lock annotations, you may use also AssertLockHeld
here and in UpdateTip
(also AcceptToMemoryPoolWithTime
, DisconnectTip
, ConnectTip
)
ACK 4e307e5 (only checked that my name is in the credits, did not look at the changes) Show signature and timestampSignature:
Timestamp of file with hash |
This change encapsulates UTXO set data within CChainState instances, removing global data `pcoinsTip` and `pcoinsviewdb`. This is necessary if we want to maintain multiple chainstates with their own rendering of the UTXO set. We introduce a class CoinsViews which consolidates the construction of a CCoins* hierarchy. Construction of its various pieces (db, coinscatcher, in-memory cache) is split up so that we avoid flushing bad state to disk if startup is interrupted. We also introduce `CChainState::CanFlushToDisk()` which tells us when it is safe to flush the chainstate based on this partial construction. This commit could be broken into smaller pieces, but it would require more ephemeral diffs to, e.g., temporarily change CCoinsViewDB's constructor invocations. Other changes: - A parameter has been added to the CCoinsViewDB constructor that allows the name of the corresponding leveldb directory to be specified. Thanks to Russell Yanofsky and Marco Falke for helpful feedback.
cdf6afb
to
c41bcbd
Compare
i.e. any CoinsViews members. Adds a lock acquisition to `gettxoutsetinfo` RPC to comply with added annotations. Co-authored-by: Russell Yanofsky <russ@yanofsky.org>
c41bcbd
to
582d2cd
Compare
Thanks for finding that stuff @MarcoFalke; fixes pushed. |
reACK 582d2cd |
ACK 582d2cd Show signature and timestampSignature:
Timestamp of file with hash |
582d2cd Cover UTXO set access with lock annotations (James O'Beirne) 5693530 refactor: have CCoins* data managed under CChainState (James O'Beirne) fae6ab6 refactor: pcoinsTip -> CChainState::CoinsTip() (James O'Beirne) Pull request description: This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11): Parent PR: #15606 Issue: #15605 Specification: https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal --- This change encapsulates UTXO set data within CChainState instances, removing global data `pcoinsTip` and `pcoinsviewdb`. This is necessary if we want to maintain multiple chainstates with their own rendering of the UTXO set. We introduce a class CoinsViews which consolidates the construction of a CCoins* hierarchy. This commit could be broken into smaller pieces, but it would require more ephemeral diffs to, e.g., temporarily change CCoinsViewDB's constructor invocations. ACKs for top commit: Sjors: reACK 582d2cd MarcoFalke: ACK 582d2cd Tree-SHA512: ec9d904fe5dca8cd2dc4b7916daa5d8bab30856dd4645987300f905e0a19f9919fce4f9d1ff03eda982943ca73e6e9a746be6cf53b46510de36e8c81a1eafba1
Similarly to upstream's -segwitheight argument for tests, we allow also controlling BIP16's activation height with -bip16height. This is used in name_multisig.py; we also include BIP16 in getblockchaininfo with the other buried softforks. Since BIP16 is not yet active in Namecoin (and thus especially not activated from the start), this is useful at least in the forseeable future. Use ::ChainstateActive().CoinsTip() instead of pcoinsTip in the Namecoin code. See bitcoin/bitcoin#16443. Rebranded the privkey / address versions in the updated descriptor_tests unit test.
…ce.cpp Summary: upcoming [[bitcoin/bitcoin#16443 | PR16443]] adds an `#include <txdb.h>` to validation.h which makes zmq want leveldb symbols. this is the first step towards avoiding that and a generally Good Thing To Do. `#include <primitives/block.h>` will do. Test Plan: ninja check check-functional Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D7550
…tate Summary: Upcoming Core [[bitcoin/bitcoin#16443 | PR16443]] adds `g_chainstate` to validation.cpp, this caused a bit of a havoc with initializaion order of pindexFinalized (also declared in validation.cpp) so I took this opportunity to encapsulate the finalizedBlockIndex pointer into CChainState additionally: * moved an assertion that cs_main lock must be held down to where the actual pointer is accessed, to guarantee that all code paths trigger that. * ~~added an assert to prevent users from changing the finalizationBlockIndex pointer to nullptr~~ Test Plan: cmake .. -DENABLE_SANITIZERS=address ninja all check check-functional Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Subscribers: deadalnix Differential Revision: https://reviews.bitcoinabc.org/D7578
Summary: Upcoming Core [[bitcoin/bitcoin#16443 | PR16443]] makes validation.h #include <txdb.h>, which has the unwelcome effect of making zmq transitively #include <leveldb/db.h> which cmake helpfully doesn't make available to it. turns out zmqpublishnotifier.cpp (the only translation unit requiring validation.h) needs it only for ReadBlockFrom disk. This looks like a good place to start to cut things out of the validation TU which is basically responsible for everything right now. This diff moves a few functions related to handling block files into the (tentatively named) readblockfromdisk.cpp translation unit so that zmqpublishnotifier.cpp no longer pulls leveldb transitively. Test Plan: ninja check check-functional Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Subscribers: deadalnix Differential Revision: https://reviews.bitcoinabc.org/D7579
Summary: This aliasing makes subsequent commits easier to review; eventually CoinsTip() will return the CCoinsViewCache managed by CChainState. bitcoin/bitcoin@fae6ab6 --- Depends on D7578 and D7579 Partial backport of Core [[bitcoin/bitcoin#16443 | PR16443]] Test Plan: ninja check check-functional Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D7580
…State Summary: This change encapsulates UTXO set data within CChainState instances, removing global data `pcoinsTip` and `pcoinsviewdb`. This is necessary if we want to maintain multiple chainstates with their own rendering of the UTXO set. We introduce a class CoinsViews which consolidates the construction of a CCoins* hierarchy. Construction of its various pieces (db, coinscatcher, in-memory cache) is split up so that we avoid flushing bad state to disk if startup is interrupted. We also introduce `CChainState::CanFlushToDisk()` which tells us when it is safe to flush the chainstate based on this partial construction. This commit could be broken into smaller pieces, but it would require more ephemeral diffs to, e.g., temporarily change CCoinsViewDB's constructor invocations. Other changes: - A parameter has been added to the CCoinsViewDB constructor that allows the name of the corresponding leveldb directory to be specified. Thanks to Russell Yanofsky and Marco Falke for helpful feedback. --- bitcoin/bitcoin@5693530 Partial backport of Core [[bitcoin/bitcoin#16443 | PR16443]] Depends on D7580 Test Plan: cmake .. -DENABLE_SANITIZERS=address ninja check check-functional Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D7581
Summary: i.e. any CoinsViews members. Adds a lock acquisition to `gettxoutsetinfo` RPC to comply with added annotations. Co-authored-by: Russell Yanofsky <russ@yanofsky.org> --- bitcoin/bitcoin@582d2cd Depends on D7581 Concludes backport of Core [[bitcoin/bitcoin#16443 | PR16443]] Test Plan: cmake .. -DENABLE_SANITIZERS=address ninja check check-functional Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D7582
This is part of the assumeutxo project:
Parent PR: #15606
Issue: #15605
Specification: https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal
This change encapsulates UTXO set data within CChainState instances, removing global data
pcoinsTip
andpcoinsviewdb
. This is necessary if we want to maintain multiple chainstates with their own rendering of the UTXO set.We introduce a class CoinsViews which consolidates the construction of a CCoins* hierarchy.
This commit could be broken into smaller pieces, but it would require more ephemeral diffs to, e.g., temporarily change CCoinsViewDB's constructor invocations.