Skip to content

Conversation

joaopaulofonseca
Copy link
Contributor

This PR implements an index of unspent transaction outputs (UTXOs).

This approach optimizes CWallet::AvailableCoins, by doing some of its work each time a wallet transaction changes. For instance, using CWallet::IsMine earlier for each transaction output, it manages to keep track only of UTXOs related to wallet addresses.

That brings large improvements on RPCs such as listunspent and fundrawtransaction. Depending on the size of your wallet and number of UTXOs, performance gains may be up to 98%.

This way, instead of going through all the wallet transactions, the search for UTXOs is made only on this updated list of outputs.

Depends on #7822 to ensure functionality behaviour remains the same.

@laanwj laanwj added the Wallet label Apr 6, 2016
@laanwj
Copy link
Member

laanwj commented Apr 6, 2016

Depending on the size of your wallet and number of UTXOs, performance gains may be up to 98%.

Yes, this makes sense.

How does it affect memory usage?

@sipa
Copy link
Member

sipa commented Apr 6, 2016

Concept ACK, it's silly that we didn't have this.

Memory usage: I expect it to consume around 80 bytes per unspent output on 64-bit platforms.

@promag
Copy link
Contributor

promag commented Apr 6, 2016

ACK

Rename first commit to something like Add index to wallet UTXO

@joaopaulofonseca
Copy link
Contributor Author

The overall memory usage will depend on the wallet size, more particularly on the UTXO count. Given the info being stored per UTXO (an outpoint and a pair with isminetype and pointer to Tx), the extra memory increment won't be that much.

@joaopaulofonseca
Copy link
Contributor Author

@promag that makes sense. This is not a cache, but more like an index..

@pedrobranco
Copy link
Contributor

Tested ACK.

In my case the improvements of listunspent are from 12s. to 0.1s .

@joaopaulofonseca joaopaulofonseca changed the title [Wallet] Add index of unspent transaction outputs (UTXOs) Add index to wallet UTXO Apr 6, 2016
@joaopaulofonseca joaopaulofonseca force-pushed the enhancement/cache-unspent-outputs branch from 7e39fb7 to f61677b Compare April 6, 2016 11:34
@jonasschnelli
Copy link
Contributor

Nice work!
Concept ACK.

@laanwj
Copy link
Member

laanwj commented Apr 6, 2016

Given the info being stored per UTXO (an outpoint and a pair with isminetype and pointer to Tx), the extra memory increment won't be that much.

So this only uses (~80 bytes) memory per actually unspent output that is in the wallet? That's great. I then don't expect the extra memory use for that to be significant. The bulk is wasted on keeping historical transactions anyway.

Concept ACK.

@joaopaulofonseca joaopaulofonseca force-pushed the enhancement/cache-unspent-outputs branch 4 times, most recently from 4939d13 to 2892f4e Compare April 6, 2016 14:49
@gmaxwell
Copy link
Contributor

gmaxwell commented Apr 6, 2016

Concept ACK. Thanks for working on this.

@promag
Copy link
Contributor

promag commented Apr 6, 2016

Solves #6573.

@instagibbs
Copy link
Member

lightly tested ACK

@laanwj laanwj mentioned this pull request Apr 15, 2016
8 tasks
@joaopaulofonseca joaopaulofonseca force-pushed the enhancement/cache-unspent-outputs branch from 2892f4e to 6fbc9bf Compare April 15, 2016 17:08
@promag
Copy link
Contributor

promag commented Apr 19, 2016

This change greatly improves the performance of AvailableCoins and indirectly listunspent and others. Is it reasonable to expect this in next 0.12 update?

@sipa
Copy link
Member

sipa commented Apr 19, 2016

@promag Only bug fixes and consensus changes are backported to old releases. It's well on track for 0.13, though.

@laanwj
Copy link
Member

laanwj commented Apr 19, 2016

I think this needs at least one more code review ACK before merge (to master, not 0.12)

@@ -498,9 +502,12 @@ bool CWallet::IsSpent(const uint256& hash, unsigned int n) const

void CWallet::AddToSpends(const COutPoint& outpoint, const uint256& wtxid)
{
// since this output is being marked as a spend output, we remove it from the UTXO index
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: spent output

@sipa
Copy link
Member

sipa commented Apr 19, 2016

How does this deal with:

  • reorganizations (coins marked as spent may go back to being unspent)?
  • conflicts (there may be multiple unconfirmed transactions at the tip, and which set of them is accepted can change easily)?
  • abandoned transactions?

@joaopaulofonseca joaopaulofonseca force-pushed the enhancement/cache-unspent-outputs branch 2 times, most recently from d12287e to 2b9c1a5 Compare April 26, 2016 09:34
@fanquake
Copy link
Member

This needs a rebase.

@joaopaulofonseca joaopaulofonseca force-pushed the enhancement/cache-unspent-outputs branch from 2b9c1a5 to 55abe3a Compare May 31, 2016 09:34
@sipa
Copy link
Member

sipa commented Jun 1, 2016

@jpdffonseca Feel like answering my questions above?

@joaopaulofonseca
Copy link
Contributor Author

@sipa I'm working to unify mapTxSpends and mapTxUnspents.

@paveljanik
Copy link
Contributor

@jpdffonseca Are you still working on this? We need performance improvements especially for large wallets.

Needs rebase.

@joaopaulofonseca
Copy link
Contributor Author

joaopaulofonseca commented Oct 10, 2016

@paveljanik lately I haven't had too much time to work on it. After testing more deeply, I realized that this solution does not always handles well some of the cases mentioned above by @sipa, so it should have a different approach.

I've been trying another improvements that optimize not the unspents list itself but other values necessary to calculate the unspents, such as the call CWallet::IsMine().
With that I obtained satisfactory results, around 50% for large wallets. I'll finish that and submit PR.

@fanquake
Copy link
Member

Closing due to inactivity. @joaopaulofonseca if you do end up finishing work on your other improvements PRs are welcome.

@joaopaulofonseca
Copy link
Contributor Author

Deprecated partially by #9939. Caching accurately these values gets a little tricky because of all the scenarios that could make the cache invalidated (conflicts, double-spends, reorganizations, abandoned transactions, etc, ...).

@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants