-
Notifications
You must be signed in to change notification settings - Fork 37.7k
Add 'sendall' RPC née sweep #24118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add 'sendall' RPC née sweep #24118
Conversation
8ca6c20
to
a2882eb
Compare
a2882eb
to
dbfdbad
Compare
Interesting idea. Can this be exposed as a testing only RPC? Or do real users need it? There are privacy implications of using sweep as well -- if it's just to be used for e.g. taking an old potentially compromised wallet and migrating to a new one is it better to do a sweep that splits into a number of normal-ish looking txns? |
Cool, concept ACK Tests look nice |
Subtract fee from amount has historically been used for sweeping and there exist some issues on this repo that illustrate that there is usage. You could achieve a "normal" looking transaction by specifying two recipients and setting an amount on one of them. There are examples to illustrate this usage in the RPC help text. |
dbfdbad
to
669c522
Compare
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
Concept ACK |
Concept ACK, I prefer this to #23534. |
Concept ACK |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK
The added sweep()
RPC function looks good. I went through the code and couldn’t find any breaking points. The same goes for the added test, which is logically sound. Kudos for this amazing work, @xekyo! 🥂
Here’s a screenshot of the displayed message when passing the command:
./src/bitcoin-cli -signet sweep
Screenshot:
The added test passed successfully, which intends that the added function works correctly. I shall do a manual test by creating a new wallet and sweeping it in some time. Meanwhile, I have some doubts and suggestions that might help make this PR even better.
The GUI will need access to this functionality too. It would be nice if the sweeping logic was refactored into a function in Edit: Actually that can wait until we figure out what the GUI is going to do for sweep. |
0f0470c
to
87e497f
Compare
I have no objection to a dedicated |
Either approach seems fine to me, but I do think it is potentially more confusing to have two completely different RPCs for sending funds: "send" vs" sweep" than to have one sending option that means "Send approximate amount. I don't care about exact amount of BTC received, and am happy if it's a little less or a little more to economize on fees and avoid change." Obviously you shouldn't use this option if you are trying to send an exact amount to someone, but it seems like it would be generally useful whenever you are sending money to one of your own wallets, or exchange accounts, or paying for any service that can be incrementally topped up. I know one of achow101's recent PR was adding more complexity to subtract from output implementation to try to do something to help sweeping, but I don't think the original (current?) semantics inherently had to add much complexity. It is true the code has been complex at different points but I think that was mostly a result of code shittiness and duplication, which have generally improved recently. |
Indeed I suspect that a |
Concept ACK I have no objections to this RPC either, but something like |
That use case is what I call generalized sweep. Sweep can be viewed as spending all UTXOs in a given list without needing to specify the output amount. Whether that list is all the UTXOs in the wallet or a specific list of some UTXOs doesn't matter. So sweep can be extended to cover your use case by allowing for inputs to be specified.
Can you describe why SFFO would be used in those cases? @xekyo and I have had discussions with many people about SFFO use cases and even though the concept of "send no more than X" usually comes up, no one can express why that behavior would ever actually be useful. Just looking through many of the issues about SFFO show that its primary use case is to sweep an entire wallet, and sometimes spend just a preset list of coins without having to calculate the fee manually. Both of these cases can be covered by a sweep function without needing to maintain SFFO in coin selection logic.
I find that SFFO makes it much harder to reason about our coin selection because we now use effective values for selection, but with SFFO, we don't actually want the effective values. |
Another SFFO use case is when your recipient is legitimately taking on the cost of business. For example, it can be used when sending a loan. |
I think, mostly, it is useful conceptually as a way to simplify creating transactions. It's a way of expressing the intention behind a transaction, and letting the implementation take care of the details, instead of being having to think about what the transaction will look like, and choose between sweep and send APIs to create it. Also the concept here is not exactly "spend no more than X." If there are two amounts associated with transaction: X is amount spent by sender, Y is amount received by receiver, then option lets you choose between fixing X and letting Y vary, or fixing Y and letting X vary. If you have 12 BTC, and want to budget exactly 1 BTC per month to spend on a service, the option let you do that. And in general, it lets you just say what your intention is instead of having to think at a lower level. Additionally, there is my main practical concern:
My medium level practical concern:
And my mini practical concern:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re-ACK 78aab64
@Sjors @stickies-v @glozow re-review this? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
code review ACK 78aab64, a few non-blocking comments/questions
_Motivation_ Currently, the wallet uses a fSubtractFeeAmount (SFFO) flag on the recipients objects for all forms of sending calls. According to the commit discussion, this flag was chiefly introduced to permit sweeping without manually calculating the fees of transactions. However, the flag leads to unintuitive behavior and makes it more complicated to test many wallet RPCs exhaustively. We proposed to introduce a dedicated `sendall` RPC with the intention to cover this functionality. Since the proposal, it was discovered in further discussion that our proposed `sendall` rpc and SFFO have subtly different scopes of operation. • sendall: Use _specific UTXOs_ to pay a destination the remainder after fees. • SFFO: Use a _specific budget_ to pay an address the remainder after fees. While `sendall` will simplify cases of spending from specific UTXOs, emptying a wallet, or burning dust, we realized that there are some cases in which SFFO is used to pay other parties from a limited budget, which can often lead to the creation of change outputs. This cannot be easily replicated using `sendall` as it would require manual computation of the appropriate change amount. As such, sendall cannot replace all uses of SFFO, but it still has a different use case and will aid in simplifying some wallet calls and numerous wallet tests. _Sendall call details_ The proposed sendall call builds a transaction from a specific subset of the wallet's UTXO pool (by default all of them) and assigns the funds to one or more receivers. Receivers can either be specified with a specific amount or receive an equal share of the remaining unassigned funds. At least one recipient must be provided without assigned amount to collect the remainder. The `sendall` call will never create change. The call has a `send_max` option that changes the default behavior of spending all UTXOs ("no UTXO left behind"), to maximizing the output amount of the transaction by skipping uneconomic UTXOs. The `send_max` option is incompatible with providing a specific set of inputs.
78aab64
to
bb84b71
Compare
Addressed all of the comments by @glozow and @MarcoFalke |
re-ACK bb84b71 |
This had 3 ACKs and I checked the range-diff. Did not review myself. |
Sorry I didn't get around to reviewing this. I'll see if I can test it later. This remains on my wish list (for a future PR):
|
Probably wouldn't hurt to also return metadata like |
bb84b71 add tests for no recipient and using send_max while inputs are specified (ishaanam) 49090ec Add sendall RPC née sweep (Murch) 902793c Extract FinishTransaction from send() (Murch) 6d2208a Extract interpretation of fee estimation arguments (Murch) a31d75e Elaborate error messages for outdated options (Murch) 35ed094 Extract prevention of outdated option names (Murch) Pull request description: Add sendall RPC née sweep _Motivation_ Currently, the wallet uses a fSubtractFeeAmount (SFFO) flag on the recipients objects for all forms of sending calls. According to the commit discussion, this flag was chiefly introduced to permit sweeping without manually calculating the fees of transactions. However, the flag leads to unintuitive behavior and makes it more complicated to test many wallet RPCs exhaustively. We proposed to introduce a dedicated `sendall` RPC with the intention to cover this functionality. Since the proposal, it was discovered in further discussion that our proposed `sendall` rpc and SFFO have subtly different scopes of operation. • sendall: Use _given UTXOs_ to pay a destination the remainder after fees. • SFFO: Use a _given budget_ to pay an address the remainder after fees. While `sendall` will simplify cases of spending a given set of UTXOs such as paying the value from one or more specific UTXOs, emptying a wallet, or burning dust, we realized that there are some cases in which SFFO is used to pay other parties from a limited budget, which can often lead to the creation of change outputs. This cannot be easily replicated using `sendall` as it would require manual computation of the appropriate change amount. As such, sendall cannot replace all uses of SFFO, but it still has a different use case and will aid in simplifying some wallet calls and numerous wallet tests. _Sendall call details_ The proposed sendall call builds a transaction from a specific subset of the wallet's UTXO pool (by default all of them) and assigns the funds to one or more receivers. Receivers can either be specified with a given amount or receive an equal share of the remaining unassigned funds. At least one recipient must be provided without assigned amount to collect the remainder. The `sendall` call will never create change. The call has a `send_max` option that changes the default behavior of spending all UTXOs ("no UTXO left behind"), to maximizing the output amount of the transaction by skipping uneconomic UTXOs. The `send_max` option is incompatible with providing a specific set of inputs. --- Edit: Replaced OP with latest commit message to reflect my updated motivation of the proposal. ACKs for top commit: achow101: re-ACK bb84b71 Tree-SHA512: 20aaf75d268cb4b144f5d6437d33ec7b5f989256b3daeeb768ae1e7f39dc6b962af8223c5cb42ecc72dc38cecd921c53c077bc0ec300b994e902412213dd2cc3
This will be reused in `sendall` so we extract it to avoid duplication. Github-Pull: bitcoin#24118 Rebased-From: 35ed094
Github-Pull: bitcoin#24118 Rebased-From: a31d75e
This will be reused in `sendall`, so we extract a method to prevent duplication. Github-Pull: bitcoin#24118 Rebased-From: 6d2208a
The final step of send either produces a PSBT or the final transaction. We extract these steps to a new helper function `FinishTransaction()` to reuse them in `sendall`. Github-Pull: bitcoin#24118 Rebased-From: 902793c
_Motivation_ Currently, the wallet uses a fSubtractFeeAmount (SFFO) flag on the recipients objects for all forms of sending calls. According to the commit discussion, this flag was chiefly introduced to permit sweeping without manually calculating the fees of transactions. However, the flag leads to unintuitive behavior and makes it more complicated to test many wallet RPCs exhaustively. We proposed to introduce a dedicated `sendall` RPC with the intention to cover this functionality. Since the proposal, it was discovered in further discussion that our proposed `sendall` rpc and SFFO have subtly different scopes of operation. • sendall: Use _specific UTXOs_ to pay a destination the remainder after fees. • SFFO: Use a _specific budget_ to pay an address the remainder after fees. While `sendall` will simplify cases of spending from specific UTXOs, emptying a wallet, or burning dust, we realized that there are some cases in which SFFO is used to pay other parties from a limited budget, which can often lead to the creation of change outputs. This cannot be easily replicated using `sendall` as it would require manual computation of the appropriate change amount. As such, sendall cannot replace all uses of SFFO, but it still has a different use case and will aid in simplifying some wallet calls and numerous wallet tests. _Sendall call details_ The proposed sendall call builds a transaction from a specific subset of the wallet's UTXO pool (by default all of them) and assigns the funds to one or more receivers. Receivers can either be specified with a specific amount or receive an equal share of the remaining unassigned funds. At least one recipient must be provided without assigned amount to collect the remainder. The `sendall` call will never create change. The call has a `send_max` option that changes the default behavior of spending all UTXOs ("no UTXO left behind"), to maximizing the output amount of the transaction by skipping uneconomic UTXOs. The `send_max` option is incompatible with providing a specific set of inputs. Github-Pull: bitcoin#24118 Rebased-From: 49090ec
Github-Pull: bitcoin#24118 Rebased-From: bb84b71
…s tx-size check cc434cb wallet: fix sendall creates tx that fails tx-size check (kouloumos) Pull request description: Fixes bitcoin/bitcoin#26011 The `sendall` RPC doesn't use `CreateTransactionInternal` as the rest of the wallet RPCs. [This has already been discussed in the original PR](bitcoin/bitcoin#24118 (comment)). By not going through that path, it never checks the transaction's weight against the maximum tx weight for transactions we're willing to relay. https://github.com/bitcoin/bitcoin/blob/447f50e4aed9a8b1d80e1891cda85801aeb80b4e/src/wallet/spend.cpp#L1013-L1018 This PR adds a check for tx-size as well as test coverage for that case. _Note: It seems that the test takes a bit of time on slower machines, I'm not sure if dropping it might be for the better._ ACKs for top commit: glozow: re ACK cc434cb via range-diff. Changes were addressing bitcoin/bitcoin#26024 (comment) and bitcoin/bitcoin#26024 (comment). achow101: ACK cc434cb w0xlt: reACK bitcoin/bitcoin@cc434cb Tree-SHA512: 64a1d8f2c737b39f3ccee90689eda1dd9c1b65f11b2c7bc0ec8bfe72f0202ce90ab4033bb0ecfc6080af8c947575059588a00938fe48e7fd553f7fb5ee03b3cc
…e check cc434cb wallet: fix sendall creates tx that fails tx-size check (kouloumos) Pull request description: Fixes bitcoin#26011 The `sendall` RPC doesn't use `CreateTransactionInternal` as the rest of the wallet RPCs. [This has already been discussed in the original PR](bitcoin#24118 (comment)). By not going through that path, it never checks the transaction's weight against the maximum tx weight for transactions we're willing to relay. https://github.com/bitcoin/bitcoin/blob/447f50e4aed9a8b1d80e1891cda85801aeb80b4e/src/wallet/spend.cpp#L1013-L1018 This PR adds a check for tx-size as well as test coverage for that case. _Note: It seems that the test takes a bit of time on slower machines, I'm not sure if dropping it might be for the better._ ACKs for top commit: glozow: re ACK cc434cb via range-diff. Changes were addressing bitcoin#26024 (comment) and bitcoin#26024 (comment). achow101: ACK cc434cb w0xlt: reACK bitcoin@cc434cb Tree-SHA512: 64a1d8f2c737b39f3ccee90689eda1dd9c1b65f11b2c7bc0ec8bfe72f0202ce90ab4033bb0ecfc6080af8c947575059588a00938fe48e7fd553f7fb5ee03b3cc
Add sendall RPC née sweep
Motivation
Currently, the wallet uses a fSubtractFeeAmount (SFFO) flag on the
recipients objects for all forms of sending calls. According to the
commit discussion, this flag was chiefly introduced to permit sweeping
without manually calculating the fees of transactions. However, the flag
leads to unintuitive behavior and makes it more complicated to test
many wallet RPCs exhaustively. We proposed to introduce a dedicated
sendall
RPC with the intention to cover this functionality.Since the proposal, it was discovered in further discussion that our
proposed
sendall
rpc and SFFO have subtly different scopes ofoperation.
• sendall:
Use given UTXOs to pay a destination the remainder after fees.
• SFFO:
Use a given budget to pay an address the remainder after fees.
While
sendall
will simplify cases of spending a given set ofUTXOs such as paying the value from one or more specific UTXOs, emptying
a wallet, or burning dust, we realized that there are some cases in
which SFFO is used to pay other parties from a limited budget,
which can often lead to the creation of change outputs. This cannot be
easily replicated using
sendall
as it would require manualcomputation of the appropriate change amount.
As such, sendall cannot replace all uses of SFFO, but it still has a
different use case and will aid in simplifying some wallet calls and
numerous wallet tests.
Sendall call details
The proposed sendall call builds a transaction from a specific
subset of the wallet's UTXO pool (by default all of them) and assigns
the funds to one or more receivers. Receivers can either be specified
with a given amount or receive an equal share of the remaining
unassigned funds. At least one recipient must be provided without
assigned amount to collect the remainder. The
sendall
call willnever create change. The call has a
send_max
option that changes thedefault behavior of spending all UTXOs ("no UTXO left behind"), to
maximizing the output amount of the transaction by skipping uneconomic
UTXOs. The
send_max
option is incompatible with providing a specificset of inputs.
Edit: Replaced OP with latest commit message to reflect my updated motivation of the proposal.