Add cursor to iterate over utxo set, use this in `gettxoutsetinfo` #7756

laanwj · 2016-03-28T16:48:09Z

I need this for something else, but I think this is simply a better API. I also vaguely remember this has been discussed before.

Add a method Cursor() to CCoinsView that returns a cursor which can be used to iterate over the whole UTXO set.
Implement GetUTXOStats in terms of this, remove GetStats() method on CCoinsView.
Change gettxoutsetinfo RPC to use new GetUTXOStats function.

This is not yet enough to break the dependency of txdb.cpp/h on main (there's CDiskTxPos to cope with, for example), but it is a step toward that (see #7815).

laanwj · 2016-03-31T09:12:35Z

Travis fail is unrelated (#7470 (comment)), respinning.

laanwj · 2016-04-06T11:05:16Z

Rebased

sipa · 2016-04-06T11:10:23Z

src/coins.cpp

@@ -45,7 +45,7 @@ bool CCoinsView::GetCoins(const uint256 &txid, CCoins &coins) const { return fal
 bool CCoinsView::HaveCoins(const uint256 &txid) const { return false; }
 uint256 CCoinsView::GetBestBlock() const { return uint256(); }
 bool CCoinsView::BatchWrite(CCoinsMap &mapCoins, const uint256 &hashBlock) { return false; }
-bool CCoinsView::GetStats(CCoinsStats &stats) const { return false; }
+CCoinsViewCursor *CCoinsView::Cursor() const { return 0; }


Using NULL would be clearer as it's a pointer.

Bjarne Stroustrup taught me not to use NULL in c++ code: http://www.stroustrup.com/bs_faq2.html#null
I don't care deeply about this though.

Interesting. Comment retracted.

sipa · 2016-04-06T11:28:37Z

rpc: Add method gettxoutsetinfo2 that uses new Cursor method

That seems outdated.

laanwj · 2016-04-06T12:02:23Z

That seems outdated.

Yes, I've removed that in the rebase, I had multiple commits with two getutxosetinfo in an intermediate step but that was annoying to rebase over #7766.

sipa · 2016-04-06T14:52:32Z

src/txdb.cpp

+
+void CCoinsViewDBCursor::Next()
+{
+    while (true) {


This should never loop more than once, I think? Seems that the loop here could hide potential problems.

Right, there can't be holes between DB_COINS records, will change this loop.

laanwj · 2016-04-09T08:14:39Z

Pushed a commit that should address @sipa's comment: get rid of loop, no more chance of hiding failures.

sipa · 2016-04-12T17:59:38Z

utACK 299fad9

Add a method Cursor() to CCoinsView that returns a cursor which can be used to iterate over the whole UTXO set. - rpc: Change gettxoutsetinfo to use new Cursor method - txdb: Remove GetStats method - Now that GetStats is implemented in terms of Cursor, remove it.

laanwj · 2016-04-15T14:33:19Z

Squashed eacf28c 299fad9 into 509cb00

…utsetinfo` 509cb00 txdb: Add Cursor() method to CCoinsView to iterate over UTXO set (Wladimir J. van der Laan)

Remove the mistaken assumption that GetKey returning false signifies an internal database issue. It will return false when the key cannot be deserialized into the (char,uint256) stanza, which indicates that the cursor has reached a different kind of key. Fixes bug bitcoin#7890 introduced in bitcoin#7756.

This builds on bitcoin#7756 and - Adds a streaming API to the HTTP server. This allows streaming data to the client chunk by chunk, which is useful when not the entire data is available at once or it is huge and wouldn't fit (efficiently) in memory. - Allows downloading the entire UTXO set through `/rest/utxoset`. This is a raw dump of all outputs, the state normally hashed by `gettxoutsetinfo`. The dump is performed in the background by making use of leveldb snapshotting, so without keeping cs_main locked. - This can be useful for analysis purposes if you don't want to mess with bitcoin core's database - Filename (via content-disposition) is `utxoset-<height>-<bestblockhash>.dat`. Also a custom `X-Best-Block` and `X-Block-Height` header is added.

…`gettxoutsetinfo` 509cb00 txdb: Add Cursor() method to CCoinsView to iterate over UTXO set (Wladimir J. van der Laan)

Remove the mistaken assumption that GetKey returning false signifies an internal database issue. It will return false when the key cannot be deserialized into the (char,uint256) stanza, which indicates that the cursor has reached a different kind of key. Fixes bug bitcoin#7890 introduced in bitcoin#7756.

Break the circular dependency between main and txdb by: - Moving `CBlockFileInfo` from `main.h` to `chain.h`. I think this makes sense, as the other block-file stuff is there too. - Moving `CDiskTxPos` from `main.h` to `txdb.h`. This type seems specific to txdb. - Pass a functor `insertBlockIndex` to `LoadBlockIndexGuts`. This leaves it up to the caller how to insert block indices. Zcash: This does not actually break the circular dependency for us yet, as we still need to pull in bitcoin/bitcoin#7756 and bitcoin/bitcoin#7904.

5147571 Break circular dependency main ↔ txdb (random-zebra) 2725fac txdb: Add Cursor() method to CCoinsView to iterate over UTXO set (random-zebra) Pull request description: Other two quick backports needed for the upcoming per-txout model. - bitcoin#7756 - Add cursor to iterate over utxo set, use this in `gettxoutsetinfo` > - Add a method `Cursor()` to `CCoinsView` that returns a cursor which can be used to iterate over the whole UTXO set. > - Implement `GetUTXOStats` in terms of this, remove `GetStats()` method on `CCoinsView`. > - Change `gettxoutsetinfo` RPC to use new `GetUTXOStats` function. - bitcoin#7815 - Break circular dependency main ↔ txdb (part of this was done already in #1302 and #1319) > - Moving `CBlockFileInfo` from main.h to chain.h. I think this makes sense, as the other block-file stuff is there too. > - Moving `CDiskTxPos` from main.h to txdb.h. This type, used for `-txindex` seems specific to txdb. > - Pass a functor `insertBlockIndex` to `LoadBlockIndexGuts`. This leaves it up to the caller how to insert block indices. ACKs for top commit: furszy: ACK 5147571 Fuzzbawls: ACK 5147571 Tree-SHA512: 166f5d797fac68667c59e002a3596af65596cd6ebee5e8b4bee538e6f66ec5f6365e4dd0bbb44be2226ebb51411d4db01f9f2417660841ed7a5125b3c8a7ad97

laanwj added UTXO Db and Indexes RPC/REST/ZMQ labels Mar 28, 2016

laanwj mentioned this pull request Mar 28, 2016

[WIP] rest: Stream entire utxo set #7759

Closed

laanwj force-pushed the 2016_03_utxo_cursor branch from ab53441 to 47d25a8 Compare March 30, 2016 07:30

laanwj mentioned this pull request Apr 5, 2016

Break circular dependency main ↔ txdb #7815

Merged

laanwj force-pushed the 2016_03_utxo_cursor branch from 47d25a8 to c1331de Compare April 6, 2016 08:46

sipa reviewed Apr 6, 2016
View reviewed changes

laanwj force-pushed the 2016_03_utxo_cursor branch from c1331de to eacf28c Compare April 6, 2016 12:08

sipa reviewed Apr 6, 2016
View reviewed changes

laanwj force-pushed the 2016_03_utxo_cursor branch from 30b7efe to 299fad9 Compare April 9, 2016 08:16

laanwj force-pushed the 2016_03_utxo_cursor branch from 299fad9 to 509cb00 Compare April 15, 2016 14:33

laanwj merged commit 509cb00 into bitcoin:master Apr 15, 2016

laanwj added a commit that referenced this pull request Apr 15, 2016

Merge #7756: Add cursor to iterate over utxo set, use this in `gettxo…

9e47fce

…utsetinfo` 509cb00 txdb: Add Cursor() method to CCoinsView to iterate over UTXO set (Wladimir J. van der Laan)

laanwj mentioned this pull request Apr 18, 2016

txdb: Fix assert crash in new UTXO set cursor #7904

Merged

laanwj mentioned this pull request May 10, 2016

Periodic automatic UTXO database backup #8037

Closed

ptschip mentioned this pull request Oct 3, 2017

dbcache phase 3 enhancements BitcoinUnlimited/BitcoinUnlimited#785

Merged

dagurval mentioned this pull request Mar 5, 2018

Add cursor to iterate over utxo set bitcoinxt/bitcoinxt#348

Merged

random-zebra mentioned this pull request Aug 9, 2020

[Refactoring] Break circular dependency main ↔ txdb PIVX-Project/PIVX#1797

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add cursor to iterate over utxo set, use this in `gettxoutsetinfo` #7756

Add cursor to iterate over utxo set, use this in `gettxoutsetinfo` #7756

Uh oh!

laanwj commented Mar 28, 2016

Uh oh!

laanwj commented Mar 31, 2016

Uh oh!

laanwj commented Apr 6, 2016

Uh oh!

sipa Apr 6, 2016

Uh oh!

laanwj Apr 6, 2016

Uh oh!

sipa Apr 6, 2016

Uh oh!

sipa commented Apr 6, 2016

Uh oh!

laanwj commented Apr 6, 2016

Uh oh!

sipa Apr 6, 2016

Uh oh!

laanwj Apr 8, 2016

Uh oh!

laanwj commented Apr 9, 2016

Uh oh!

sipa commented Apr 12, 2016

Uh oh!

laanwj commented Apr 15, 2016

Uh oh!

Uh oh!

Add cursor to iterate over utxo set, use this in gettxoutsetinfo #7756

Add cursor to iterate over utxo set, use this in gettxoutsetinfo #7756

Uh oh!

Conversation

laanwj commented Mar 28, 2016

Uh oh!

laanwj commented Mar 31, 2016

Uh oh!

laanwj commented Apr 6, 2016

Uh oh!

sipa Apr 6, 2016

Choose a reason for hiding this comment

Uh oh!

laanwj Apr 6, 2016

Choose a reason for hiding this comment

Uh oh!

sipa Apr 6, 2016

Choose a reason for hiding this comment

Uh oh!

sipa commented Apr 6, 2016

Uh oh!

laanwj commented Apr 6, 2016

Uh oh!

sipa Apr 6, 2016

Choose a reason for hiding this comment

Uh oh!

laanwj Apr 8, 2016

Choose a reason for hiding this comment

Uh oh!

laanwj commented Apr 9, 2016

Uh oh!

sipa commented Apr 12, 2016

Uh oh!

laanwj commented Apr 15, 2016

Uh oh!

Uh oh!

Add cursor to iterate over utxo set, use this in `gettxoutsetinfo` #7756

Add cursor to iterate over utxo set, use this in `gettxoutsetinfo` #7756