-
Notifications
You must be signed in to change notification settings - Fork 37.7k
wallet: Faster transaction creation by removing pre-set-inputs fetching responsibility from Coin Selection #25685
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
wallet: Faster transaction creation by removing pre-set-inputs fetching responsibility from Coin Selection #25685
Conversation
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
2545eab
to
8cf6caf
Compare
508f3c5
to
84b4ffe
Compare
rebased, conflicts solved. |
Concept ACK, this seems like a good improvement. |
It's not clear to me that this provides any significant benefit over a much simpler change of just excluding pre-selected inputs from AvailableCoins. The only other improvement this provides is better error reporting, but I think it would be better to just go through SelectCoins and improve its error reporting instead of pulling things out so that errors are reported elsewhere. |
There are several benefits, (aside from what I wrote in the PR description): Sources Architecture:
Tangible Improvements:
We are currently failing inside the Coin Selection process due:
These errors have nothing to do with the Coin Selection process. Why keep them there? It's not that I located these errors on a random place. I placed them inside a "Fetch Coins" procedure whose main responsibility is fetch the coins that the user pre-selected and fetch the available coins in the wallet. Which seems totally reasonable for me. I mean, this errors have been obscured inside Coin Selection all this time, not being properly notified, because they should had never been placed there in the first place. It's all about a proper responsibility division to make the process clearer and cleaner so it's less error-prone (and in this case, in some occasions, actually faster and less resource consuming). |
To make the speed improvements more visible, pushed a new benchmark f6d0bb2. Basically, the new bench measures the transaction creation process, for a wallet with ~250k UTXO, only using the pre-selected-inputs inside coin control. Setting This is the result on this PR (tip f6d0bb2):
vs This is the result on current master (tip 4a4289e):
SummaryThe benchmark took to run in master: 96.37 milliseconds, while in this PR: 1 millisecond 🚀 . Updated PR description with the benchmark results. |
0eb720f
to
f6d0bb2
Compare
79f3b6f
to
7b7b77e
Compare
Took me a bit but, while was tackling the feedback, ended up re-organizing the commits changes better. The rationale for each modification should be clearer now. What did was:
|
Goal 1: Benchmark the transaction creation process for pre-selected-inputs only. Setting `m_allow_other_inputs=false` to disallow the wallet to include coins automatically. Goal 2: Benchmark the transaction creation process for pre-selected-inputs and coin selection. ----------------------- Benchmark Setup: 1) Generates a 5k blockchain, loading the wallet with 5k transactions with two outputs each. 2) Fetch 4 random UTXO from the wallet's available coins and pre-select them as inputs inside CoinControl. Benchmark (Goal 1): Call `CreateTransaction` providing the coin control, who has set `m_allow_other_inputs=false` and the manually selected coins. Benchmark (Goal 2): Call `CreateTransaction` providing the coin control, who has set `m_allow_other_inputs=true` and the manually selected coins.
5bea640
to
3fcb545
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
reACK 3fcb545
Code Review ACK 3fcb545 |
ACK 3fcb545 |
Due to bitcoin/bitcoin#25685, we now have upstream logic for specifying pre-selected inputs in wallet/spend.cpp. This allows us to simplify the Namecoin-specific logic for spending with a predefined input (the previous name coin).
…ed total amount 7362f8e refactor: make CoinsResult total amounts members private (furszy) 3282fad wallet: add assert to SelectionResult::Merge for safety (S3RK) c4e3b7d wallet: SelectCoins, return early if wallet's UTXOs cannot cover the target (furszy) cac2725 test: bugfix, coinselector_test, use 'CoinsResult::Erase/Add' instead of direct member access (furszy) cf79384 test: Coin Selection, duplicated preset inputs selection (furszy) 341ba7f test: wallet, coverage for CoinsResult::Erase function (furszy) f930aef wallet: bugfix, 'CoinsResult::Erase' is erasing only one output of the set (furszy) Pull request description: This comes with #26559. Solving few bugs inside the wallet's transaction creation process and adding test coverage for them. Plus, making use of the `CoinsResult::total_amount` cached value inside the Coin Selection process to return early if we don't have enough funds to cover the target amount. ### Bugs 1) The `CoinsResult::Erase` method removes only one output from the available coins vector (there is a [loop break](https://github.com/bitcoin/bitcoin/blob/c1061be14a515b0ed4f4d646fcd0378c62e6ded3/src/wallet/spend.cpp#L112) that should have never been there) and not all the preset inputs. Which on master is not a problem, because since [#25685](bitcoin/bitcoin#25685) we are no longer using the method. But, it's a bug on v24 (check [#26559](bitcoin/bitcoin#26559)). This method it's being fixed and not removed because I'm later using it to solve another bug inside this PR. 2) As we update the total cached amount of the `CoinsResult` object inside `AvailableCoins` and we don't use such function inside the coin selection tests (we manually load up the `CoinsResult` object), there is a discrepancy between the outputs that we add/erase and the total amount cached value. ### Improvements * This makes use of the `CoinsResult` total amount field to early return with an "Insufficient funds" error inside Coin Selection if the tx target amount is greater than the sum of all the wallet available coins plus the preset inputs amounts (we don't need to perform the entire coin selection process if we already know that there aren't enough funds inside our wallet). ### Test Coverage 1) Adds test coverage for the duplicated preset input selection bug that we have in v24. Where the wallet invalidly selects the preset inputs twice during the Coin Selection process. Which ends up with a "good" Coin Selection result that does not cover the total tx target amount. Which, alone, crashes the wallet due an insane fee. But.. to make it worst, adding the subtract fee from output functionality to this mix ends up with the wallet by-passing the "insane" fee assertion, decreasing the output amount to fulfill the insane fee, and.. sadly, broadcasting the tx to the network. 2) Adds test coverage for the `CoinsResult::Erase` method. ------------------------------------ TO DO: * [ ] Update [#26559 ](bitcoin/bitcoin#26559) description. ACKs for top commit: achow101: ACK 7362f8e glozow: ACK 7362f8e, I assume there will be a followup PR to add coin selection sanity checks and we can discuss the best way to do that there. josibake: ACK [7362f8e](bitcoin/bitcoin@7362f8e) Tree-SHA512: 37a6828ea10d8d36c8d5873ceede7c8bef72ae4c34bef21721fa9dad83ad6dba93711c3170a26ab6e05bdbc267bb17433da08ccb83b82956d05fb16090328cba
… amount 7362f8e refactor: make CoinsResult total amounts members private (furszy) 3282fad wallet: add assert to SelectionResult::Merge for safety (S3RK) c4e3b7d wallet: SelectCoins, return early if wallet's UTXOs cannot cover the target (furszy) cac2725 test: bugfix, coinselector_test, use 'CoinsResult::Erase/Add' instead of direct member access (furszy) cf79384 test: Coin Selection, duplicated preset inputs selection (furszy) 341ba7f test: wallet, coverage for CoinsResult::Erase function (furszy) f930aef wallet: bugfix, 'CoinsResult::Erase' is erasing only one output of the set (furszy) Pull request description: This comes with bitcoin#26559. Solving few bugs inside the wallet's transaction creation process and adding test coverage for them. Plus, making use of the `CoinsResult::total_amount` cached value inside the Coin Selection process to return early if we don't have enough funds to cover the target amount. ### Bugs 1) The `CoinsResult::Erase` method removes only one output from the available coins vector (there is a [loop break](https://github.com/bitcoin/bitcoin/blob/c1061be14a515b0ed4f4d646fcd0378c62e6ded3/src/wallet/spend.cpp#L112) that should have never been there) and not all the preset inputs. Which on master is not a problem, because since [bitcoin#25685](bitcoin#25685) we are no longer using the method. But, it's a bug on v24 (check [bitcoin#26559](bitcoin#26559)). This method it's being fixed and not removed because I'm later using it to solve another bug inside this PR. 2) As we update the total cached amount of the `CoinsResult` object inside `AvailableCoins` and we don't use such function inside the coin selection tests (we manually load up the `CoinsResult` object), there is a discrepancy between the outputs that we add/erase and the total amount cached value. ### Improvements * This makes use of the `CoinsResult` total amount field to early return with an "Insufficient funds" error inside Coin Selection if the tx target amount is greater than the sum of all the wallet available coins plus the preset inputs amounts (we don't need to perform the entire coin selection process if we already know that there aren't enough funds inside our wallet). ### Test Coverage 1) Adds test coverage for the duplicated preset input selection bug that we have in v24. Where the wallet invalidly selects the preset inputs twice during the Coin Selection process. Which ends up with a "good" Coin Selection result that does not cover the total tx target amount. Which, alone, crashes the wallet due an insane fee. But.. to make it worst, adding the subtract fee from output functionality to this mix ends up with the wallet by-passing the "insane" fee assertion, decreasing the output amount to fulfill the insane fee, and.. sadly, broadcasting the tx to the network. 2) Adds test coverage for the `CoinsResult::Erase` method. ------------------------------------ TO DO: * [ ] Update [bitcoin#26559 ](bitcoin#26559) description. ACKs for top commit: achow101: ACK 7362f8e glozow: ACK 7362f8e, I assume there will be a followup PR to add coin selection sanity checks and we can discuss the best way to do that there. josibake: ACK [7362f8e](bitcoin@7362f8e) Tree-SHA512: 37a6828ea10d8d36c8d5873ceede7c8bef72ae4c34bef21721fa9dad83ad6dba93711c3170a26ab6e05bdbc267bb17433da08ccb83b82956d05fb16090328cba
Only a GUI issue, for the "use available balance" button when the user manually select inputs on the send screen. The previous behavior for getAvailableBalance when the coin control has selected inputs was to return the sum of them. Instead, we are currently returning the wallet's available total balance minus the selected coins total amount. Reason: We missed to update the GetAvailableBalance function to include the coin control selected coins on bitcoin#25685. Context: Since bitcoin#25685 we skip the selected coins inside `AvailableCoins`, the reason is that there is no need to walk through the entire wallet's txes map just to get coins that could have gotten by just doing a simple mapWallet.find).
Only a GUI issue, for the "use available balance" button when the user manually select inputs on the send screen. The previous behavior for getAvailableBalance when the coin control has selected inputs was to return the sum of them. Instead, we are currently returning the wallet's available total balance minus the selected coins total amount. Reason: We missed to update the GetAvailableBalance function to include the coin control selected coins on bitcoin#25685. Context: Since bitcoin#25685 we skip the selected coins inside `AvailableCoins`, the reason is that there is no need to walk through the entire wallet's txes map just to get coins that could have gotten by just doing a simple mapWallet.find).
Only a GUI issue, for the "use available balance" button when the user manually select inputs on the send screen. The previous behavior for getAvailableBalance when the coin control has selected inputs was to return the sum of them. Instead, we are currently returning the wallet's available total balance minus the selected coins total amount. Reason: We missed to update the GetAvailableBalance function to include the coin control selected coins on bitcoin#25685. Context: Since bitcoin#25685 we skip the selected coins inside `AvailableCoins`, the reason is that there is no need to walk through the entire wallet's txes map just to get coins that could have gotten by just doing a simple mapWallet.find).
Only a GUI issue, for the "use available balance" button when the user manually select inputs on the send screen. The previous behavior for getAvailableBalance when the coin control has selected inputs was to return the sum of them. Instead, we are currently returning the wallet's available total balance minus the selected coins total amount. Reason: We missed to update the GetAvailableBalance function to include the coin control selected coins on bitcoin#25685. Context: Since bitcoin#25685 we skip the selected coins inside `AvailableCoins`, the reason is that there is no need to walk through the entire wallet's txes map just to get coins that could have gotten by just doing a simple mapWallet.find).
Only a GUI issue, for the "use available balance" button when the user manually select inputs on the send screen. The previous behavior for getAvailableBalance when the coin control has selected inputs was to return the sum of them. Instead, we are currently returning the wallet's available total balance minus the selected coins total amount. Reason: We missed to update the GetAvailableBalance function to include the coin control selected coins on bitcoin#25685. Context: Since bitcoin#25685 we skip the selected coins inside `AvailableCoins`, the reason is that there is no need to walk through the entire wallet's txes map just to get coins that could have gotten by just doing a simple mapWallet.find).
Only a GUI issue, for the "use available balance" button when the user manually select inputs on the send screen. The previous behavior for getAvailableBalance when the coin control has selected inputs was to return the sum of them. Instead, we are currently returning the wallet's available total balance minus the selected coins total amount. Reason: We missed to update the GetAvailableBalance function to include the coin control selected coins on bitcoin#25685. Context: Since bitcoin#25685 we skip the selected coins inside `AvailableCoins`, the reason is that there is no need to walk through the entire wallet's txes map just to get coins that could have gotten by just doing a simple mapWallet.find).
The previous behavior for getAvailableBalance when coin control has selected coins was to return the sum of them. Instead, we are currently returning the wallet's available total balance minus the selected coins total amount. This turns into a GUI-only issue for the "use available balance" button when the user manually select coins in the send screen. Reason: We missed to update the GetAvailableBalance function to include the coin control selected coins on bitcoin#25685. Context: Since bitcoin#25685 we skip the selected coins inside `AvailableCoins`, the reason is that there is no need to traverse the wallet's txes map just to get coins that can directly be fetched by their id.
The previous behavior for getAvailableBalance when coin control has selected coins was to return the sum of them. Instead, we are currently returning the wallet's available total balance minus the selected coins total amount. This turns into a GUI-only issue for the "use available balance" button when the user manually select coins in the send screen. Reason: We missed to update the GetAvailableBalance function to include the coin control selected coins on bitcoin#25685. Context: Since bitcoin#25685 we skip the selected coins inside `AvailableCoins`, the reason is that there is no need to traverse the wallet's txes map just to get coins that can directly be fetched by their id.
The previous behavior for getAvailableBalance when coin control has selected coins was to return the sum of them. Instead, we are currently returning the wallet's available total balance minus the selected coins total amount. This turns into a GUI-only issue for the "use available balance" button when the user manually select coins in the send screen. Reason: We missed to update the GetAvailableBalance function to include the coin control selected coins on bitcoin#25685. Context: Since bitcoin#25685 we skip the selected coins inside `AvailableCoins`, the reason is that there is no need to traverse the wallet's txes map just to get coins that can directly be fetched by their id.
… coins 68eed5d test,gui: add coverage for PSBT creation on legacy watch-only wallets (furszy) 306aab5 test,gui: decouple widgets and model into a MiniGui struct (furszy) 2f76ac0 test,gui: decouple chain and wallet initialization from test case (furszy) cd98b71 gui: 'getAvailableBalance', include watch only balance (furszy) 74eac3a test: add coverage for 'useAvailableBalance' functionality (furszy) dc1cc1c gui: bugfix, getAvailableBalance skips selected coins (furszy) Pull request description: Fixes bitcoin-core/gui#688 and #26687. First Issue Description (bitcoin-core/gui#688): The previous behavior for `getAvailableBalance`, when the coin control had selected coins, was to return the sum of them. Instead, we are currently returning the wallet's available total balance minus the selected coins total amount. Reason: Missed to update the `GetAvailableBalance` function to include the coin control selected coins on #25685. Context: Since #25685 we skip the selected coins inside `AvailableCoins`, the reason is that there is no need to waste resources walking through the entire wallet's txes map just to get coins that could have gotten by just doing a simple `mapWallet.find`). Places Where This Generates Issues (only when the user manually select coins via coin control): 1) The GUI balance check prior the transaction creation process. 2) The GUI "useAvailableBalance" functionality. Note 1: As the GUI uses a balance cache since bitcoin-core/gui#598, this issue does not affect the regular spending process. Only arises when the user manually select coins. Note 2: Added test coverage for the `useAvailableBalance` functionality. ---------------------------------- Second Issue Description (#26687): As we are using a cached balance on `WalletModel::getAvailableBalance`, the function needs to include the watch-only available balance for wallets with private keys disabled. ACKs for top commit: Sjors: tACK 68eed5d achow101: ACK 68eed5d theStack: ACK 68eed5d Tree-SHA512: 674f3e050024dabda2ff4a04b9ed3750cf54a040527204c920e1e38bd3d7f5fd4d096e4fd08a0fea84ee6abb5070f022b5c0d450c58fd30202ef05ebfd7af6d3
The previous behavior for getAvailableBalance when coin control has selected coins was to return the sum of them. Instead, we are currently returning the wallet's available total balance minus the selected coins total amount. This turns into a GUI-only issue for the "use available balance" button when the user manually select coins in the send screen. Reason: We missed to update the GetAvailableBalance function to include the coin control selected coins on bitcoin#25685. Context: Since bitcoin#25685 we skip the selected coins inside `AvailableCoins`, the reason is that there is no need to traverse the wallet's txes map just to get coins that can directly be fetched by their id.
# Context (Current Flow on Master)
In the transaction creation process, in order to select which coins the new transaction will spend,
we first obtain all the available coins known by the wallet, which means walking-through the
wallet txes map, gathering the ones that fulfill certain spendability requirements in a vector.
This coins vector is then provided to the Coin Selection process, which first checks if the user
has manually selected any input (which could be internal, aka known by the wallet, or external),
and if it does, it fetches them by searching each of them inside the wallet and/or inside the
Coin Control external tx data.
Then, after finding the pre-selected-inputs and gathering them in a vector, the Coin Selection
process walks-through the entire available coins vector once more just to erase coins that are
in both vectors. So the Coin Selection process doesn’t pick them twice (duplicate inputs inside
the same transaction).
# Process Workflow Changes
Now, a new method,
FetchCoins
will be responsible for:Which will occur prior to the Coin Selection process. Which allows us to never include the
pre-selected-inputs inside the available coins vector in the first place, as well as doing other
nice improvements (written below).
So, Coin Selection can perform its main responsibility without mixing it with having to fetch
internal/external coins nor any slow and unneeded duplicate coins verification.
# Summarizing the Improvements:
If any pre-selected-input lookup fail, the process will return the error right away.
(before, the wallet was fetching all the wallet available coins, walking through the
entire txes map, and then failing for an invalid pre-selected-input inside SelectCoins)
The pre-selected-inputs lookup failure causes are properly described on the return error.
(before, we were returning an "Insufficient Funds" error for everything, even if the failure
was due a not solvable external input)
Faster Coin Selection: no longer need to "remove the pre-set inputs from the available coins
vector so that Coin Selection doesn't pick them" (which meant to loop-over the entire
available coins vector at Coin Selection time, erasing duplicate coins that were pre-selected).
Now, the available coins vector, which is built after the pre-selected-inputs fetching,
doesn’t include the already selected inputs in the first place.
Faster transaction creation for transactions that only use manually selected inputs.
We now will return early, as soon as we finish fetching the pre-selected-inputs and
not perform the resources expensive calculation of walking-through the entire wallet
txes map to obtain the available coins (coins that we will not use).
Added a new bench (f6d0bb2) measuring the transaction creation process, for a wallet with ~250k UTXO, only using the pre-selected-inputs inside coin control. Setting
m_allow_other_inputs=false
to disallow the wallet to include coins automatically.Result on this PR (tip f6d0bb2):
WalletCreateTransaction
vs
Result on master (tip 4a4289e):
WalletCreateTransaction
The benchmark took to run in master: 96.37 milliseconds, while in this PR: 1 millisecond 🚀 .