Skip to content

Conversation

jonatack
Copy link
Member

@jonatack jonatack commented Mar 27, 2020

This PR implements point 2 from #17314 (show balance per wallet):

  • create an RPC that returns the ismine.trusted balance for all loaded wallets
  • update bitcoin-cli -getinfo to use the RPC

before

$ bitcoin-cli -getinfo -regtest
{
  "version": 199900,
  "blocks": 15599,
  "headers": 15599,
  "verificationprogress": 1,
  "timeoffset": 0,
  "connections": 0,
  "proxy": "",
  "difficulty": 4.656542373906925e-10,
  "chain": "regtest",
  "keypoolsize": 1000,
  "paytxfee": 0.00000000,
  "balance": 0.00001000,
  "relayfee": 0.00001000
}

after

$ bitcoin-cli -getinfo -regtest
{
  "version": 199900,
  "blocks": 15599,
  "headers": 15599,
  "verificationprogress": 1,
  "timeoffset": 0,
  "connections": 0,
  "proxy": "",
  "difficulty": 4.656542373906925e-10,
  "chain": "regtest",
  "balances": {
    "": 0.00001000,
    "Encrypted": 0.00003500,
    "day-to-day": 0.00000120,
    "side project": 0.00000094
  },
  "relayfee": 0.00001000
}

Review club discussion about this PR is here: https://bitcoincore.reviews/18453.

This approach is simpler, adds no additional rpc calls in -getinfo and runs faster than doing it on the client side inside bitcoin-cli.cpp, but it also adds the first multiwallet RPC and would need to be well-considered in terms of the API.

@laanwj
Copy link
Member

laanwj commented Mar 28, 2020

Concept ACK

@jonasschnelli
Copy link
Contributor

Adds one more call to getinfo. Under the hood, it loads the same structures GetBalance().m_mine_trusted. Not efficient, but since performance is acceptable, I think it's worth the change to prevent from future issues and allows stripping down getwalletinfo

utACK f73ace0

@laanwj
Copy link
Member

laanwj commented Mar 28, 2020

What would make me really happy, when you're making changes here anyway, is implementing point 2 from #17314: show balance per wallet. This is much more useful in the multi-wallet case.

@promag
Copy link
Contributor

promag commented Mar 28, 2020

Concept ACK.

Agree in making @laanwj happy. The change requires a listwallets call before the batch.

@jonatack
Copy link
Member Author

Good idea, will do.

@laanwj
Copy link
Member

laanwj commented Mar 28, 2020

The change requires a listwallets call before the batch

I don't think that's necessary, getbalances will already retrieve the balances for all currently active wallets, which is what I mean, It doesn't need to list inactive ones in the datadir IMO.
Oh this is not true, I'm confused apparently.

@jonatack
Copy link
Member Author

Something like this? (getbalances version)

$ bitcoin-cli -getinfo -regtest
{
  "version": 199900,
  "blocks": 15599,
  "headers": 15599,
  "verificationprogress": 1,
  "timeoffset": 0,
  "connections": 0,
  "proxy": "",
  "difficulty": 4.656542373906925e-10,
  "chain": "regtest",
  "balances": [
    {
      "": {
        "mine": {
          "trusted": 0.00001000,
          "untrusted_pending": 0.00000000,
          "immature": 0.00000000
        }
      }
    },
    {
      "encrypted": {
        "mine": {
          "trusted": 0.00003500,
          "untrusted_pending": 0.00000000,
          "immature": 0.00000000
        }
      }
    },
  ],
  "relayfee": 0.00001000
}

or like? (mine.trusted balances only)

$ bitcoin-cli -getinfo -regtest
{
  "version": 199900,
  "blocks": 15599,
  "headers": 15599,
  "verificationprogress": 1,
  "timeoffset": 0,
  "connections": 0,
  "proxy": "",
  "difficulty": 4.656542373906925e-10,
  "chain": "regtest",
  "balances": [
    {
      "name": "",
      "amount": 0.00001000
    },
    {
      "name": "encrypted",
      "amount": 0.00003500
    }
  ],
  "relayfee": 0.00001000
}

@laanwj
Copy link
Member

laanwj commented Mar 28, 2020

It's best if -getinfo fits on a single screen, so I'd prefer a really condensed form, for ex.

{
  "version": 199900,
  "blocks": 15599,
  "headers": 15599,
  "verificationprogress": 1,
  "timeoffset": 0,
  "connections": 0,
  "proxy": "",
  "difficulty": 4.656542373906925e-10,
  "chain": "regtest",
  "balances": {
    "": 0.00001000,
    "encrypted": 0.00003500,
  },
  "relayfee": 0.00001000
}

@jonatack
Copy link
Member Author

It's best if -getinfo fits on a single screen, so I'd prefer a really condensed form

Perfect -- thanks 👍

@jonatack jonatack force-pushed the call-getbalances-for-getinfo-balance branch from f73ace0 to 11676f9 Compare March 29, 2020 00:45
@jonatack jonatack changed the title cli: enable -getinfo to fetch wallet balance from getbalances() cli -getinfo: enable multiwallet balances and no longer depend on getwalletinfo balance Mar 29, 2020
@jonatack jonatack force-pushed the call-getbalances-for-getinfo-balance branch from 11676f9 to 0da1a8f Compare March 29, 2020 01:12
@jonatack
Copy link
Member Author

Done. If this approach is ok, I'll add tests for the new call.

@promag
Copy link
Contributor

promag commented Mar 29, 2020

Why not iterate on the cli?

@jonatack
Copy link
Member Author

I could be wrong, but this seemed cleaner, and I think we'll ultimately want either an rpc like this or allow passing a multi bool to getbalance or getbalances to do the same.

@jonatack jonatack force-pushed the call-getbalances-for-getinfo-balance branch from 0da1a8f to 6a2f1db Compare March 29, 2020 13:13
@promag
Copy link
Contributor

promag commented Mar 29, 2020

Not sure about the approach TBH. I'd rather see the cli use existing calls.

The new RPC getwalletbalances is the first to use multiple wallets and could lead to more of multiwallet RPC. For instance, getaddressesbylabel could return all address of all loaded wallets.

@jonatack jonatack force-pushed the call-getbalances-for-getinfo-balance branch from 6a2f1db to a5ea571 Compare March 29, 2020 15:04
@jonatack
Copy link
Member Author

jonatack commented Mar 29, 2020

This works, but I understand. Interested in feedback. Note that for now getwalletbalances doesn't show up in the help. Rebased to add tests for getwalletbalances in the wallet_multiwallet tests (which also improves the existing tests).

@sipa
Copy link
Member

sipa commented Apr 1, 2020

I'm not opposed to having "across-all-wallets" RPCs, but perhaps it's worth making them visually distinct? How about "getallwalletbalances" or "allwalletsgetbalance"?

@jonatack
Copy link
Member Author

jonatack commented Apr 1, 2020

Yes. I was debating internally about "listbalances"...

@fjahr
Copy link
Contributor

fjahr commented Apr 3, 2020

ACK a5ea571

Reviewed code and did some light testing locally.

I think multi wallet RPCs will be useful for a lot of users. For naming I don't have strong feelings but since "multi wallet" is already the term most frequently used for this scenario verbally, I think I would prefer if that gets used if people think the distinction is necessary. So I would prefer something like "(get)multiwalletbalances" to something like listbalances or something with allwallets.

Copy link
Contributor

@robot-visions robot-visions left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK. Code change looks great.

Local testing:

  • Unit tests passed
  • Functional tests passed (interface_bitcoin_cli.py and wallet_multiwallet.py)
  • Manual smoke test passed (created multiple wallets, generated coins, made sure bitcoin-cli -regtest -getinfo and bitcoin-cli -regtest getwalletbalances behaved as expected)

Thanks for hosting the next PR Review Club! During the session, I'd also be curious to learn when it's appropriate to add new RPCs to the bitcoin-cli documentation.

@DrahtBot
Copy link
Contributor

DrahtBot commented Apr 5, 2020

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Copy link
Contributor

@vasild vasild left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approach ACK.

The modified tests test/functional/interface_bitcoin_cli.py and test/functional/wallet_multiwallet.py pass on my computer and the output of bitcoin-cli -getinfo -regtest looks as intended.

Comment on lines +730 to +717
auto locked_chain = wallet.chain().lock();
LOCK(wallet.cs_wallet);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These locks are acquired inside GetBalance(), so they can be removed from here?

A wallet's chain will never change and, I assume, all wallets' chains are the same. That is - we are operating on one chain here, for all wallets. Can we / should we lock that chain before the for-loop in order to get a consistent view (i.e. avoid getting one wallet's balance as of height H and another wallet's balance as of height H+1)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A wallet's chain will never change and, I assume, all wallets' chains are the same. That is - we are operating on one chain here, for all wallets. Can we / should we lock that chain before the for-loop in order to get a consistent view (i.e. avoid getting one wallet's balance as of height H and another wallet's balance as of height H+1)?

We generally don't want wallet code to lock up the node, and after #16426 we are removing the ability of wallet code to be able to do this. It synchronization between different wallets is a concern, a solution could be to return tip hashes alongside balances. Or to restart the wallet loop if the tip changes midway.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. So only the first question above remains (remove the locks acquisition).

{
UniValue obj{UniValue::VOBJ};
for (const std::shared_ptr<CWallet>& rpc_wallet : GetWallets()) {
if (!EnsureWalletIsAvailable(rpc_wallet.get(), request.fHelp)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be doing any of this if request.fHelp is true? It happens when bitcoin-cli help is executed even though none of its output contains getwalletbalances or any balances.

@@ -4264,6 +4286,7 @@ static const CRPCCommand commands[] =
{ "wallet", "encryptwallet", &encryptwallet, {"passphrase"} },
{ "wallet", "getaddressesbylabel", &getaddressesbylabel, {"label"} },
{ "wallet", "getaddressinfo", &getaddressinfo, {"address"} },
{ "wallet", "getwalletbalances", &getwalletbalances, {} },
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you plan to document this before merge? If "no", then are there other undocumented commands?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the consensus is to add a public RPC, yes, it should have documentation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, makes sense. IMO the current approach where the iterate-over-wallets logic is inside bitcoind and exposed via RPC is better than having that logic inside bitcoin-cli because in the latter case it cannot be reused by RPC users - should they need that functionality they would have to replicate the logic inside their applications.

@jonasschnelli
Copy link
Contributor

Concept ACK.
I added the RPC label as this now adds a new PRC command (should probably be in the PR description).

I strongly agree with @sipa. Across-all-wallets calls should have a naming convention. IMO allwallet[s] should be the required string (works with get, list, execute, etc.). Once we have started adding across-all-wallets calls it will be difficult to change the concept (so its worth investing some brainpower in how that should work).

Another approach would be to add a special RPC endpoint like /allwallets that will combine the JSON output of each per-wallet-output (more a generic RPC multiwallet approach).
From the users perspective that could look like:

bitcoin-cli -allwallets getbalances
Output:
[
  {"name": "mywallet",
  {"output": 
		{
		  "mine": {
		    "trusted": 0.00000000,
		    "untrusted_pending": 0.00000000,
		    "immature": 0.00000000
		  }
		}
	}
,
  {"name": "encryptedwallet",
  {"output": 
		{
		  "mine": {
		    "trusted": 1.00000000,
		    "untrusted_pending": 0.00000000,
		    "immature": 0.00000000
		  }
		}
	}
,
...
]

I'm not convince that this is the better approach (has probably some performance and flexibility downsides), just wanted to highlight the importantness of defining the first across-all-wallets RPC.

@jonatack jonatack force-pushed the call-getbalances-for-getinfo-balance branch from f9391c3 to a96508e Compare April 16, 2020 17:59
@jonatack jonatack force-pushed the call-getbalances-for-getinfo-balance branch 2 times, most recently from 98e37e0 to d8177fd Compare April 17, 2020 00:28
@jonatack jonatack force-pushed the call-getbalances-for-getinfo-balance branch from d8177fd to d720f7f Compare April 20, 2020 15:13
@jb55
Copy link
Contributor

jb55 commented Apr 20, 2020

@jonasschnelli:

Therefore I think it would make sense to support some sort of batching option that allow across-wallets batches (for performance reasons).

ACK this. Right now I have bash scripts for doing multi-wallet things like balances and coins but it's pretty horrible and slow since I can't batch requests. It would be so much nicer if there was a special wallet endpoint you could hit which signals that you want the result from all wallets, or at least a way to batch requests for different wallets.

@jonatack jonatack force-pushed the call-getbalances-for-getinfo-balance branch from d720f7f to c414b42 Compare April 21, 2020 13:09
@jonatack
Copy link
Member Author

Rebased on master for the CI fix.

Copy link
Member

@luke-jr luke-jr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept NACK specifically to enabling access to the wallet list (or non-selected wallets) for existing interfaces.

Up until now, RPCs would only ever access the wallet the request was sent to...

@DrahtBot
Copy link
Contributor

🐙 This pull request conflicts with the target branch and needs rebase.

@jonatack
Copy link
Member Author

jonatack commented Apr 25, 2020

Thanks to everyone for reviewing.

Summary:
4 Concept ACKs
1 Approach ACK
1 Concept NACK
2 Approach NACKs
4 ACKs

To take the feedback into account, a client-side, smaller-scope version of this feature for -getinfo only, with Concept ACKs, ACKs, and several rounds of review, is ready now for final review at #18594. Edit: #18594 is merged, thanks everyone.

@jonatack jonatack closed this Apr 25, 2020
meshcollider added a commit that referenced this pull request May 23, 2020
5edad5c test: add -getinfo multiwallet functional tests (Jon Atack)
903b6c1 rpc: drop unused JSONRPCProcessBatchReply size arg, refactor (Jon Atack)
afce85e cli: use GetWalletBalances() functionality for -getinfo (Jon Atack)
9f01849 cli: create GetWalletBalances() to fetch multiwallet balances (Jon Atack)
7430775 cli: lift -rpcwallet logic up to CommandLineRPC() (Jon Atack)
29f2cbd cli: extract connection exception handler, -rpcwait logic (Jon Atack)

Pull request description:

  This PR is a client-side version of #18453, per review feedback there and [review club discussions](https://bitcoincore.reviews/18453#meeting-log). It updates `bitcoin-cli -getinfo` on the client side to display wallet name and balance for the loaded wallets when more than one is loaded (e.g. you are in "multiwallet mode") and `-rpcwallet=` is not passed; otherwise, behavior is unchanged.

  before
  ```json
  $ bitcoin-cli -getinfo -regtest
  {
    "version": 199900,
    "blocks": 15599,
    "headers": 15599,
    "verificationprogress": 1,
    "timeoffset": 0,
    "connections": 0,
    "proxy": "",
    "difficulty": 4.656542373906925e-10,
    "chain": "regtest",
    "balance": 0.00001000,
    "relayfee": 0.00001000
  }

  ```
  after
  ```json
  $ bitcoin-cli -getinfo -regtest
  {
    "version": 199900,
    "blocks": 15599,
    "headers": 15599,
    "verificationprogress": 1,
    "timeoffset": 0,
    "connections": 0,
    "proxy": "",
    "difficulty": 4.656542373906925e-10,
    "chain": "regtest",
    "balances": {
      "": 0.00001000,
      "Encrypted": 0.00003500,
      "day-to-day": 0.00000120,
      "side project": 0.00000094
    }
  }
  ```
  -----

  `Review club` discussion about this PR is here: https://bitcoincore.reviews/18453

  This PR can be manually tested by building, creating/loading/unloading several wallets with `bitcoin-cli createwallet/loadwallet/unloadwallet` and running `bitcoin-cli -getinfo` and `bitcoin-cli -rpcwallet=<wallet-name> -getinfo`.

  `wallet_multiwallet.py --usecli` provides regression test coverage on this change, along with `interface_bitcoin_cli.py` where this PR adds test coverage.

  Credit to Wladimir J. van der Laan for the idea in #17314 and #18453 (comment).

ACKs for top commit:
  promag:
    Tested ACK 5edad5c.
  jnewbery:
    utACK 5edad5c
  meshcollider:
    Code review ACK 5edad5c

Tree-SHA512: 4ca36c5f6c49936b40afb605c44459c1d5b80b5bd84df634007ca276b3f6c102a0cb382f9d528370363ee32c94b0d7ffa15184578eaf8de74179e566c5c5cee5
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request May 25, 2020
5edad5c test: add -getinfo multiwallet functional tests (Jon Atack)
903b6c1 rpc: drop unused JSONRPCProcessBatchReply size arg, refactor (Jon Atack)
afce85e cli: use GetWalletBalances() functionality for -getinfo (Jon Atack)
9f01849 cli: create GetWalletBalances() to fetch multiwallet balances (Jon Atack)
7430775 cli: lift -rpcwallet logic up to CommandLineRPC() (Jon Atack)
29f2cbd cli: extract connection exception handler, -rpcwait logic (Jon Atack)

Pull request description:

  This PR is a client-side version of bitcoin#18453, per review feedback there and [review club discussions](https://bitcoincore.reviews/18453#meeting-log). It updates `bitcoin-cli -getinfo` on the client side to display wallet name and balance for the loaded wallets when more than one is loaded (e.g. you are in "multiwallet mode") and `-rpcwallet=` is not passed; otherwise, behavior is unchanged.

  before
  ```json
  $ bitcoin-cli -getinfo -regtest
  {
    "version": 199900,
    "blocks": 15599,
    "headers": 15599,
    "verificationprogress": 1,
    "timeoffset": 0,
    "connections": 0,
    "proxy": "",
    "difficulty": 4.656542373906925e-10,
    "chain": "regtest",
    "balance": 0.00001000,
    "relayfee": 0.00001000
  }

  ```
  after
  ```json
  $ bitcoin-cli -getinfo -regtest
  {
    "version": 199900,
    "blocks": 15599,
    "headers": 15599,
    "verificationprogress": 1,
    "timeoffset": 0,
    "connections": 0,
    "proxy": "",
    "difficulty": 4.656542373906925e-10,
    "chain": "regtest",
    "balances": {
      "": 0.00001000,
      "Encrypted": 0.00003500,
      "day-to-day": 0.00000120,
      "side project": 0.00000094
    }
  }
  ```
  -----

  `Review club` discussion about this PR is here: https://bitcoincore.reviews/18453

  This PR can be manually tested by building, creating/loading/unloading several wallets with `bitcoin-cli createwallet/loadwallet/unloadwallet` and running `bitcoin-cli -getinfo` and `bitcoin-cli -rpcwallet=<wallet-name> -getinfo`.

  `wallet_multiwallet.py --usecli` provides regression test coverage on this change, along with `interface_bitcoin_cli.py` where this PR adds test coverage.

  Credit to Wladimir J. van der Laan for the idea in bitcoin#17314 and bitcoin#18453 (comment).

ACKs for top commit:
  promag:
    Tested ACK 5edad5c.
  jnewbery:
    utACK 5edad5c
  meshcollider:
    Code review ACK 5edad5c

Tree-SHA512: 4ca36c5f6c49936b40afb605c44459c1d5b80b5bd84df634007ca276b3f6c102a0cb382f9d528370363ee32c94b0d7ffa15184578eaf8de74179e566c5c5cee5
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.