Skip to content

Conversation

murchandamus
Copy link
Contributor

@murchandamus murchandamus commented Jan 20, 2022

Add sendall RPC née sweep

Motivation
Currently, the wallet uses a fSubtractFeeAmount (SFFO) flag on the
recipients objects for all forms of sending calls. According to the
commit discussion, this flag was chiefly introduced to permit sweeping
without manually calculating the fees of transactions. However, the flag
leads to unintuitive behavior and makes it more complicated to test
many wallet RPCs exhaustively. We proposed to introduce a dedicated
sendall RPC with the intention to cover this functionality.

Since the proposal, it was discovered in further discussion that our
proposed sendall rpc and SFFO have subtly different scopes of
operation.
• sendall:
Use given UTXOs to pay a destination the remainder after fees.
• SFFO:
Use a given budget to pay an address the remainder after fees.

While sendall will simplify cases of spending a given set of
UTXOs such as paying the value from one or more specific UTXOs, emptying
a wallet, or burning dust, we realized that there are some cases in
which SFFO is used to pay other parties from a limited budget,
which can often lead to the creation of change outputs. This cannot be
easily replicated using sendall as it would require manual
computation of the appropriate change amount.

As such, sendall cannot replace all uses of SFFO, but it still has a
different use case and will aid in simplifying some wallet calls and
numerous wallet tests.

Sendall call details
The proposed sendall call builds a transaction from a specific
subset of the wallet's UTXO pool (by default all of them) and assigns
the funds to one or more receivers. Receivers can either be specified
with a given amount or receive an equal share of the remaining
unassigned funds. At least one recipient must be provided without
assigned amount to collect the remainder. The sendall call will
never create change. The call has a send_max option that changes the
default behavior of spending all UTXOs ("no UTXO left behind"), to
maximizing the output amount of the transaction by skipping uneconomic
UTXOs. The send_max option is incompatible with providing a specific
set of inputs.


Edit: Replaced OP with latest commit message to reflect my updated motivation of the proposal.

@murchandamus murchandamus marked this pull request as ready for review January 20, 2022 22:25
@JeremyRubin
Copy link
Contributor

Interesting idea. Can this be exposed as a testing only RPC? Or do real users need it?

There are privacy implications of using sweep as well -- if it's just to be used for e.g. taking an old potentially compromised wallet and migrating to a new one is it better to do a sweep that splits into a number of normal-ish looking txns?

@jamesob
Copy link
Contributor

jamesob commented Jan 20, 2022

Cool, concept ACK

Tests look nice

@murchandamus
Copy link
Contributor Author

Subtract fee from amount has historically been used for sweeping and there exist some issues on this repo that illustrate that there is usage.

You could achieve a "normal" looking transaction by specifying two recipients and setting an amount on one of them. There are examples to illustrate this usage in the RPC help text.

@DrahtBot
Copy link
Contributor

DrahtBot commented Jan 21, 2022

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #22751 (rpc/wallet: add simulaterawtransaction RPC by kallewoof)
  • #21576 (rpc, gui: bumpfee signer support by Sjors)
  • #21283 (Implement BIP 370 PSBTv2 by achow101)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@brunoerg
Copy link
Contributor

Concept ACK

@glozow
Copy link
Member

glozow commented Jan 21, 2022

Concept ACK, I prefer this to #23534.

@w0xlt
Copy link
Contributor

w0xlt commented Jan 22, 2022

Concept ACK

Copy link
Contributor

@shaavan shaavan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK

The added sweep() RPC function looks good. I went through the code and couldn’t find any breaking points. The same goes for the added test, which is logically sound. Kudos for this amazing work, @xekyo! 🥂

Here’s a screenshot of the displayed message when passing the command:

./src/bitcoin-cli -signet sweep

Screenshot:

PR

The added test passed successfully, which intends that the added function works correctly. I shall do a manual test by creating a new wallet and sweeping it in some time. Meanwhile, I have some doubts and suggestions that might help make this PR even better.

@achow101
Copy link
Member

achow101 commented Jan 24, 2022

The GUI will need access to this functionality too. It would be nice if the sweeping logic was refactored into a function in src/wallet/spend.cpp so that the GUI can access it.

Edit: Actually that can wait until we figure out what the GUI is going to do for sweep.

@Sjors
Copy link
Member

Sjors commented Jan 28, 2022

I have no objection to a dedicated sweep RPC call, but I don't agree with the premise that subtracting fee from output is only useful for sweeping a full wallet. See #24142 (comment)

@ryanofsky
Copy link
Contributor

I have no objection to a dedicated sweep RPC call, but I don't agree with the premise that subtracting fee from output is only useful for sweeping a full wallet. See #24142 (comment)

Either approach seems fine to me, but I do think it is potentially more confusing to have two completely different RPCs for sending funds: "send" vs" sweep" than to have one sending option that means "Send approximate amount. I don't care about exact amount of BTC received, and am happy if it's a little less or a little more to economize on fees and avoid change." Obviously you shouldn't use this option if you are trying to send an exact amount to someone, but it seems like it would be generally useful whenever you are sending money to one of your own wallets, or exchange accounts, or paying for any service that can be incrementally topped up.

I know one of achow101's recent PR was adding more complexity to subtract from output implementation to try to do something to help sweeping, but I don't think the original (current?) semantics inherently had to add much complexity. It is true the code has been complex at different points but I think that was mostly a result of code shittiness and duplication, which have generally improved recently.

@Sjors
Copy link
Member

Sjors commented Jan 28, 2022

Indeed I suspect that a sweep RPC will be useless if we decide to keep 'subtracting fee from output' functionality. In that case adding a sweep feature to the send RPC would make more sense. C-lightling has a special case amount all for that purpose.

@w0xlt
Copy link
Contributor

w0xlt commented Jan 28, 2022

Concept ACK

I have no objections to this RPC either, but something like bitcoin-cli -named sendtoaddress address="...." sweep=true might be simpler for users instead of two commands to send funds.

@achow101
Copy link
Member

I have no objection to a dedicated sweep RPC call, but I don't agree with the premise that subtracting fee from output is only useful for sweeping a full wallet. See #24142 (comment)

That use case is what I call generalized sweep. Sweep can be viewed as spending all UTXOs in a given list without needing to specify the output amount. Whether that list is all the UTXOs in the wallet or a specific list of some UTXOs doesn't matter. So sweep can be extended to cover your use case by allowing for inputs to be specified.

but it seems like it would be generally useful whenever you are sending money to one of your own wallets, or exchange accounts, or paying for any service that can be incrementally topped up.

Can you describe why SFFO would be used in those cases? @xekyo and I have had discussions with many people about SFFO use cases and even though the concept of "send no more than X" usually comes up, no one can express why that behavior would ever actually be useful. Just looking through many of the issues about SFFO show that its primary use case is to sweep an entire wallet, and sometimes spend just a preset list of coins without having to calculate the fee manually. Both of these cases can be covered by a sweep function without needing to maintain SFFO in coin selection logic.

I know one of achow101's recent PR was adding more complexity to subtract from output implementation to try to do something to help sweeping, but I don't think the original (current?) semantics inherently had to add much complexity. It is true the code has been complex at different points but I think that was mostly a result of code shittiness and duplication, which have generally improved recently.

I find that SFFO makes it much harder to reason about our coin selection because we now use effective values for selection, but with SFFO, we don't actually want the effective values.

@luke-jr
Copy link
Member

luke-jr commented Jan 29, 2022

Another SFFO use case is when your recipient is legitimately taking on the cost of business. For example, it can be used when sending a loan.

@ryanofsky
Copy link
Contributor

ryanofsky commented Jan 29, 2022

but it seems like it would be generally useful whenever you are sending money to one of your own wallets, or exchange accounts, or paying for any service that can be incrementally topped up.

Can you describe why SFFO would be used in those cases? @xekyo and I have had discussions with many people about SFFO use cases and even though the concept of "send no more than X" usually comes up, no one can express why that behavior would ever actually be useful.

I think, mostly, it is useful conceptually as a way to simplify creating transactions. It's a way of expressing the intention behind a transaction, and letting the implementation take care of the details, instead of being having to think about what the transaction will look like, and choose between sweep and send APIs to create it.

Also the concept here is not exactly "spend no more than X." If there are two amounts associated with transaction: X is amount spent by sender, Y is amount received by receiver, then option lets you choose between fixing X and letting Y vary, or fixing Y and letting X vary. If you have 12 BTC, and want to budget exactly 1 BTC per month to spend on a service, the option let you do that. And in general, it lets you just say what your intention is instead of having to think at a lower level.

Additionally, there is my main practical concern:

  • This is duplicating existing CreateTransaction(Internal) logic instead of just calling CreateTransaction with the right options. The claim is that this will be a code simplification, but the followup PR Deprecate SubtractFeeFromOutputs #24142 is +588/−148 lines. And this is while REMOVING features which I think we agree are useful: providing GUI support for sweeping, and being able to select coins to sweep. The new code and tests are going to grow and become even more complex after adding these features.

My medium level practical concern:

  • Deprecating subtract from output in Deprecate SubtractFeeFromOutputs #24142 presumably is going to break existing workflows, and there doesn't seem to be a good release notes or FAQ style item saying what that the problem is with subtract from output, why its removal is justified, and how to transition to the sweep API.

And my mini practical concern:

  • This sweep API might be less safe than send APIs because it doesn't force you to specify amount you are trying to send. It's easier to fat-finger by accidentally sweeping the wrong wallet (or when manual coin selection is added) sweeping the wrong coin and spending an amount larger than you intended. Maybe this is not a very big concern. If it is a concern, it could also be addresed by adding a mandatory amount argument requiring you to specify total amount of the sweep or an override value like "unchecked" or "yolo"

Copy link
Contributor

@w0xlt w0xlt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK 78aab64

@achow101
Copy link
Member

@Sjors @stickies-v @glozow re-review this?

Copy link
Member

@glozow glozow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code review ACK 78aab64, a few non-blocking comments/questions

murchandamus and others added 2 commits March 29, 2022 16:37
_Motivation_
Currently, the wallet uses a fSubtractFeeAmount (SFFO) flag on the
recipients objects for all forms of sending calls. According to the
commit discussion, this flag was chiefly introduced to permit sweeping
without manually calculating the fees of transactions. However, the flag
leads to unintuitive behavior and makes it more complicated to test
many wallet RPCs exhaustively. We proposed to introduce a dedicated
`sendall` RPC with the intention to cover this functionality.

Since the proposal, it was discovered in further discussion that our
proposed `sendall` rpc and SFFO have subtly different scopes of
operation.
• sendall:
  Use _specific UTXOs_ to pay a destination the remainder after fees.
• SFFO:
  Use a _specific budget_ to pay an address the remainder after fees.

While `sendall` will simplify cases of spending from specific UTXOs,
emptying a wallet, or burning dust, we realized that there are some
cases in which SFFO is used to pay other parties from a limited budget,
which can often lead to the creation of change outputs. This cannot be
easily replicated using `sendall` as it would require manual computation
of the appropriate change amount.

As such, sendall cannot replace all uses of SFFO, but it still has a
different use case and will aid in simplifying some wallet calls and
numerous wallet tests.

_Sendall call details_
The proposed sendall call builds a transaction from a specific subset of
the wallet's UTXO pool (by default all of them) and assigns the funds to
one or more receivers. Receivers can either be specified with a specific
amount or receive an equal share of the remaining unassigned funds. At
least one recipient must be provided without assigned amount to collect
the remainder. The `sendall` call will never create change. The call has
a `send_max` option that changes the default behavior of spending all
UTXOs ("no UTXO left behind"), to maximizing the output amount of the
transaction by skipping uneconomic UTXOs. The `send_max` option is
incompatible with providing a specific set of inputs.
@murchandamus
Copy link
Contributor Author

Addressed all of the comments by @glozow and @MarcoFalke

@achow101
Copy link
Member

re-ACK bb84b71

@fanquake fanquake requested review from glozow, stickies-v and Sjors March 30, 2022 07:43
@maflcko
Copy link
Member

maflcko commented Mar 30, 2022

This had 3 ACKs and I checked the range-diff. Did not review myself.

@maflcko maflcko merged commit f4e5d70 into bitcoin:master Mar 30, 2022
@Sjors
Copy link
Member

Sjors commented Mar 30, 2022

Sorry I didn't get around to reviewing this. I'll see if I can test it later.

This remains on my wish list (for a future PR):

a new RPC call that returns a concise list of outpoints with their amount, like so:

16a9...97f:7 0.000100
34fe...e8d:0 0.00200

We could even use transaction short ids, if there's no duplicate in any given wallet (return the full id if there is).

@maflcko
Copy link
Member

maflcko commented Mar 30, 2022

Probably wouldn't hurt to also return metadata like depth etc..

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Apr 3, 2022
bb84b71 add tests for no recipient and using send_max while inputs are specified (ishaanam)
49090ec Add sendall RPC née sweep (Murch)
902793c Extract FinishTransaction from send() (Murch)
6d2208a Extract interpretation of fee estimation arguments (Murch)
a31d75e Elaborate error messages for outdated options (Murch)
35ed094 Extract prevention of outdated option names (Murch)

Pull request description:

  Add sendall RPC née sweep

  _Motivation_
  Currently, the wallet uses a fSubtractFeeAmount (SFFO) flag on the
  recipients objects for all forms of sending calls. According to the
  commit discussion, this flag was chiefly introduced to permit sweeping
  without manually calculating the fees of transactions. However, the flag
  leads to unintuitive behavior and makes it more complicated to test
  many wallet RPCs exhaustively. We proposed to introduce a dedicated
  `sendall` RPC with the intention to cover this functionality.

  Since the proposal, it was discovered in further discussion that our
  proposed `sendall` rpc and SFFO have subtly different scopes of
  operation.
  • sendall:
    Use _given UTXOs_ to pay a destination the remainder after fees.
  • SFFO:
    Use a _given budget_ to pay an address the remainder after fees.

  While `sendall` will simplify cases of spending a given set of
  UTXOs such as paying the value from one or more specific UTXOs, emptying
  a wallet, or burning dust, we realized that there are some cases in
  which SFFO is used to pay other parties from a limited budget,
  which can often lead to the creation of change outputs. This cannot be
  easily replicated using `sendall` as it would require manual
  computation of the appropriate change amount.

  As such, sendall cannot replace all uses of SFFO, but it still has a
  different use case and will aid in simplifying some wallet calls and
  numerous wallet tests.

  _Sendall call details_
  The proposed sendall call builds a transaction from a specific
  subset of the wallet's UTXO pool (by default all of them) and assigns
  the funds to one or more receivers. Receivers can either be specified
  with a given amount or receive an equal share of the remaining
  unassigned funds. At least one recipient must be provided without
  assigned amount to collect the remainder. The `sendall` call will
  never create change. The call has a `send_max` option that changes the
  default behavior of spending all UTXOs ("no UTXO left behind"), to
  maximizing the output amount of the transaction by skipping uneconomic
  UTXOs. The `send_max` option is incompatible with providing a specific
  set of inputs.

  ---
  Edit: Replaced OP with latest commit message to reflect my updated motivation of the proposal.

ACKs for top commit:
  achow101:
    re-ACK bb84b71

Tree-SHA512: 20aaf75d268cb4b144f5d6437d33ec7b5f989256b3daeeb768ae1e7f39dc6b962af8223c5cb42ecc72dc38cecd921c53c077bc0ec300b994e902412213dd2cc3
luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request May 21, 2022
This will be reused in `sendall` so we extract it to avoid
duplication.

Github-Pull: bitcoin#24118
Rebased-From: 35ed094
luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request May 21, 2022
luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request May 21, 2022
This will be reused in `sendall`, so we extract a method to prevent
duplication.

Github-Pull: bitcoin#24118
Rebased-From: 6d2208a
luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request May 21, 2022
The final step of send either produces a PSBT or the final transaction.
We extract these steps to a new helper function `FinishTransaction()` to
reuse them in `sendall`.

Github-Pull: bitcoin#24118
Rebased-From: 902793c
luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request May 21, 2022
_Motivation_
Currently, the wallet uses a fSubtractFeeAmount (SFFO) flag on the
recipients objects for all forms of sending calls. According to the
commit discussion, this flag was chiefly introduced to permit sweeping
without manually calculating the fees of transactions. However, the flag
leads to unintuitive behavior and makes it more complicated to test
many wallet RPCs exhaustively. We proposed to introduce a dedicated
`sendall` RPC with the intention to cover this functionality.

Since the proposal, it was discovered in further discussion that our
proposed `sendall` rpc and SFFO have subtly different scopes of
operation.
• sendall:
  Use _specific UTXOs_ to pay a destination the remainder after fees.
• SFFO:
  Use a _specific budget_ to pay an address the remainder after fees.

While `sendall` will simplify cases of spending from specific UTXOs,
emptying a wallet, or burning dust, we realized that there are some
cases in which SFFO is used to pay other parties from a limited budget,
which can often lead to the creation of change outputs. This cannot be
easily replicated using `sendall` as it would require manual computation
of the appropriate change amount.

As such, sendall cannot replace all uses of SFFO, but it still has a
different use case and will aid in simplifying some wallet calls and
numerous wallet tests.

_Sendall call details_
The proposed sendall call builds a transaction from a specific subset of
the wallet's UTXO pool (by default all of them) and assigns the funds to
one or more receivers. Receivers can either be specified with a specific
amount or receive an equal share of the remaining unassigned funds. At
least one recipient must be provided without assigned amount to collect
the remainder. The `sendall` call will never create change. The call has
a `send_max` option that changes the default behavior of spending all
UTXOs ("no UTXO left behind"), to maximizing the output amount of the
transaction by skipping uneconomic UTXOs. The `send_max` option is
incompatible with providing a specific set of inputs.

Github-Pull: bitcoin#24118
Rebased-From: 49090ec
luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request May 21, 2022
achow101 added a commit to bitcoin-core/gui that referenced this pull request Sep 15, 2022
…s tx-size check

cc434cb wallet: fix sendall creates tx that fails tx-size check (kouloumos)

Pull request description:

  Fixes bitcoin/bitcoin#26011

  The `sendall` RPC doesn't use `CreateTransactionInternal` as the rest of
  the wallet RPCs. [This has already been discussed in the original PR](bitcoin/bitcoin#24118 (comment)).
  By not going through that path, it never checks the transaction's weight
  against the maximum tx weight for transactions we're willing to relay.
  https://github.com/bitcoin/bitcoin/blob/447f50e4aed9a8b1d80e1891cda85801aeb80b4e/src/wallet/spend.cpp#L1013-L1018
  This PR adds a check for tx-size as well as test coverage for that case.

  _Note: It seems that the test takes a bit of time on slower machines,
  I'm not sure if dropping it might be for the better._

ACKs for top commit:
  glozow:
    re ACK cc434cb via range-diff. Changes were addressing bitcoin/bitcoin#26024 (comment) and bitcoin/bitcoin#26024 (comment).
  achow101:
    ACK cc434cb
  w0xlt:
    reACK bitcoin/bitcoin@cc434cb

Tree-SHA512: 64a1d8f2c737b39f3ccee90689eda1dd9c1b65f11b2c7bc0ec8bfe72f0202ce90ab4033bb0ecfc6080af8c947575059588a00938fe48e7fd553f7fb5ee03b3cc
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Sep 15, 2022
…e check

cc434cb wallet: fix sendall creates tx that fails tx-size check (kouloumos)

Pull request description:

  Fixes bitcoin#26011

  The `sendall` RPC doesn't use `CreateTransactionInternal` as the rest of
  the wallet RPCs. [This has already been discussed in the original PR](bitcoin#24118 (comment)).
  By not going through that path, it never checks the transaction's weight
  against the maximum tx weight for transactions we're willing to relay.
  https://github.com/bitcoin/bitcoin/blob/447f50e4aed9a8b1d80e1891cda85801aeb80b4e/src/wallet/spend.cpp#L1013-L1018
  This PR adds a check for tx-size as well as test coverage for that case.

  _Note: It seems that the test takes a bit of time on slower machines,
  I'm not sure if dropping it might be for the better._

ACKs for top commit:
  glozow:
    re ACK cc434cb via range-diff. Changes were addressing bitcoin#26024 (comment) and bitcoin#26024 (comment).
  achow101:
    ACK cc434cb
  w0xlt:
    reACK bitcoin@cc434cb

Tree-SHA512: 64a1d8f2c737b39f3ccee90689eda1dd9c1b65f11b2c7bc0ec8bfe72f0202ce90ab4033bb0ecfc6080af8c947575059588a00938fe48e7fd553f7fb5ee03b3cc
@bitcoin bitcoin locked and limited conversation to collaborators Mar 30, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.