Skip to content

Conversation

S3RK
Copy link
Contributor

@S3RK S3RK commented Mar 22, 2021

Rationale: make it possible to backup your wallet with listdescriptors command

  • The default behaviour is still to show public version
  • For private version only the root xprv is returned

Example use-case:

> bitcoin-cli -regtest -named createwallet wallet_name=old descriptors=true
> bitcoin-cli -regtest -rpcwallet=old listdescriptors true | jq '.descriptors' > descriptors.txt

> bitcoin-cli -regtest -named createwallet wallet_name=new descriptors=true blank=true
> bitcoin-cli -regtest -rpcwallet=new importdescriptors "$(cat descriptors.txt)"

In case of watch-only wallet without private keys there will be following output:

error code: -4
error message:
Can't get descriptor string.

@DrahtBot
Copy link
Contributor

DrahtBot commented Mar 22, 2021

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

@DrahtBot
Copy link
Contributor

DrahtBot commented May 3, 2021

🕵️ @achow101 @sipa have been requested to review this pull request as specified in the REVIEWERS file.

@@ -1736,7 +1736,9 @@ RPCHelpMan listdescriptors()
return RPCHelpMan{
"listdescriptors",
"\nList descriptors imported into a descriptor-enabled wallet.",
{},
{
{"private", RPCArg::Type::BOOL, /* default */ "false", "Show private descriptors."}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this should be a numbered parameter.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you saying it shouldn't be a parameter at all? Or it should be a "named-only" parameter? I can't see how to make it "named-only"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Named-only parameters are provided in the positional API as an "options" Object.

@@ -1736,7 +1736,9 @@ RPCHelpMan listdescriptors()
return RPCHelpMan{
"listdescriptors",
"\nList descriptors imported into a descriptor-enabled wallet.",
{},
{
{"private", RPCArg::Type::BOOL, /* default */ "false", "Show private descriptors."}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO it would be better to have a separate RPC method for private data.

@achow101
Copy link
Member

achow101 commented Jun 3, 2021

Concept ACK. The second commit seems unrelated.

@S3RK
Copy link
Contributor Author

S3RK commented Jun 24, 2021

Fixed silent merge conflict and split the 2nd commit into #22334

@S3RK S3RK force-pushed the listdescriptors_private branch from d59e43b to a0d9919 Compare July 6, 2021 07:49
@S3RK S3RK force-pushed the listdescriptors_private branch from a0d9919 to 2854ddc Compare July 7, 2021 06:41
@ghost
Copy link

ghost commented Jul 7, 2021

Tried listdescriptors RPC although I have no clue what are these 6 descriptors.

Create Wallet:

POST / HTTP/1.1
Host: 127.0.0.1:18777
Authorization: Basic hObGNqTTZjR0Z6YzNkd
Content-Type: text/plain
Content-Length: 106

{"jsonrpc": "1.0", "id": "curltest", "method": "createwallet", "params": ["DT1",false,false,"",true,true]}

List descriptors:


POST /wallet/DT1 HTTP/1.1
Host: 127.0.0.1:18777
Authorization: Basic hObGNqTTZjR0Z6YzNkd
Content-Type: text/plain
Content-Length: 83

{"jsonrpc": "1.0", "id": "curltest", "method": "listdescriptors", "params": [true]}

Response (I get the same response for true and false in params:

{
    "result": {
        "wallet_name": "DT1",
        "descriptors": [
            {
                "desc": "pkh([07f2e700/44'/1'/0']tpubDC89pDwHmvJV6xtEEWniQYEZ9Nga9cshwTHAT3dHfyWftFUtUWoQ5qAr4scvE853a2rFLqH4ZhoJtNDhSB8HS9oG8KDN5rWCaFotTFmKSAV/0/*)#ek3c29xk",
                "timestamp": 1625657174,
                "active": true,
                "internal": false,
                "range": [
                    0,
                    999
                ],
                "next": 0
            },
            {
                "desc": "wpkh([07f2e700/84'/1'/0']tpubDD49UXKp4mBeHhZKURhTFqhnnuMu5EA88v5GWFm3wnQJfZmyMwSsQqM6X3eFUwkAcRD4j9wiwoGzmphJMwEB1PwdwQuTbPdKUjhmpEsfqmh/0/*)#07l98jpn",
                "timestamp": 1625657175,
                "active": true,
                "internal": false,
                "range": [
                    0,
                    999
                ],
                "next": 0
            },
            {
                "desc": "pkh([07f2e700/44'/1'/0']tpubDC89pDwHmvJV6xtEEWniQYEZ9Nga9cshwTHAT3dHfyWftFUtUWoQ5qAr4scvE853a2rFLqH4ZhoJtNDhSB8HS9oG8KDN5rWCaFotTFmKSAV/1/*)#gz5ehskw",
                "timestamp": 1625657175,
                "active": true,
                "internal": true,
                "range": [
                    0,
                    999
                ],
                "next": 0
            },
            {
                "desc": "sh(wpkh([07f2e700/49'/1'/0']tpubDDMzDzQabs6VaDCvGa8jxNPZuDp28No6UBzathahL63xj3zWndRnNzTGqrCRSNpJDSZNnhkEKq2GwPfyGMkGip9Kqq4UkqE5Jb2x9EBsMzM/0/*))#kx6a8xhn",
                "timestamp": 1625657174,
                "active": true,
                "internal": false,
                "range": [
                    0,
                    999
                ],
                "next": 0
            },
            {
                "desc": "sh(wpkh([07f2e700/49'/1'/0']tpubDDMzDzQabs6VaDCvGa8jxNPZuDp28No6UBzathahL63xj3zWndRnNzTGqrCRSNpJDSZNnhkEKq2GwPfyGMkGip9Kqq4UkqE5Jb2x9EBsMzM/1/*))#r85tlezv",
                "timestamp": 1625657175,
                "active": true,
                "internal": true,
                "range": [
                    0,
                    999
                ],
                "next": 0
            },
            {
                "desc": "wpkh([07f2e700/84'/1'/0']tpubDD49UXKp4mBeHhZKURhTFqhnnuMu5EA88v5GWFm3wnQJfZmyMwSsQqM6X3eFUwkAcRD4j9wiwoGzmphJMwEB1PwdwQuTbPdKUjhmpEsfqmh/1/*)#726y683t",
                "timestamp": 1625657175,
                "active": true,
                "internal": true,
                "range": [
                    0,
                    999
                ],
                "next": 0
            }
        ]
    },
    "error": null,
    "id": "curltest"
}

@achow101
Copy link
Member

achow101 commented Jul 7, 2021

Response (I get the same response for true and false in params:

Are you sure you are using this branch? I get the private key with true, and the normalized public descriptor with false.

@achow101
Copy link
Member

achow101 commented Jul 7, 2021

ACK 2854ddc

Reviewed code and tested manually.

@ghost
Copy link

ghost commented Jul 7, 2021

Are you sure you are using this branch? I get the private key with true, and the normalized public descriptor with false.

Yes just noticed the difference. false returns tpub and true returns tprv

Sorry they looked same and both had 6 results. So, I was confused.

@S3RK
Copy link
Contributor Author

S3RK commented Jul 7, 2021

@prayank23 Thanks for testing! By default descriptor wallet has two descriptors for each type:

  1. Legacy — pkh(...)
  2. Wrapped SegWit — sh(wpkh(...))
  3. Native SegWit, bech32 — wpkh(...)

One of the two descriptors is for receiving, marked with internal: false and one for change internal: true.

So in the end you've got 6 of them. Each descriptor is derived from the same extended master private key (you can see it's fingerprint repeating in the descriptors; 07f2e700 in your example) using corresponding derivation paths. See BIPs: 44, 49 and 84.

Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tACK 2854ddc

@Rspigler
Copy link
Contributor

Rspigler commented Jul 7, 2021

I'm getting a build error trying to test:

(edited out).

I'm getting these on master as well, so investigating/removing from here as it is not this PR specific.

Copy link
Member

@jonatack jonatack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 2854ddc

Some minor suggestions, happy to re-ACK if you update.

@S3RK S3RK force-pushed the listdescriptors_private branch from 2854ddc to cd36796 Compare July 10, 2021 13:20
@S3RK S3RK force-pushed the listdescriptors_private branch from cd36796 to bb822a7 Compare July 10, 2021 13:21
@S3RK
Copy link
Contributor Author

S3RK commented Jul 10, 2021

Added a test case for the error when trying to get private descriptors for watch-only wallet. Plus minor readability and grammar fixes

Copy link
Member

@jonatack jonatack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the test!

ACK bb822a7 per git diff 2854ddc bb822a7

Verified the test assertions added in this pull fail on master as expected, e.g. they have to be commented-out for the test file to pass

diff --git a/test/functional/wallet_listdescriptors.py b/test/functional/wallet_listdescriptors.py
index 6e7be929cc..61de4621bf 100755
--- a/test/functional/wallet_listdescriptors.py
+++ b/test/functional/wallet_listdescriptors.py
@@ -71,7 +71,7 @@ class ListDescriptorsTest(BitcoinTestFramework):
             ],
         }
         assert_equal(expected, wallet.listdescriptors())
-        assert_equal(expected, wallet.listdescriptors(False))
+        # assert_equal(expected, wallet.listdescriptors(False))
 
         self.log.info('Test list private descriptors')
         expected_private = {
@@ -84,16 +84,16 @@ class ListDescriptorsTest(BitcoinTestFramework):
                  'next': 0},
             ],
         }
-        assert_equal(expected_private, wallet.listdescriptors(True))
+        # assert_equal(expected_private, wallet.listdescriptors(True))
 
         self.log.info("Test listdescriptors with encrypted wallet")
         wallet.encryptwallet("pass")
         assert_equal(expected, wallet.listdescriptors())
 
         self.log.info('Test list private descriptors with encrypted wallet')
-        assert_raises_rpc_error(-13, 'Please enter the wallet passphrase with walletpassphrase first.', wallet.listdescriptors, True)
+        # assert_raises_rpc_error(-13, 'Please enter the wallet passphrase with walletpassphrase first.', wallet.listdescriptors, True)
         wallet.walletpassphrase(passphrase="pass", timeout=1000000)
-        assert_equal(expected_private, wallet.listdescriptors(True))
+        # assert_equal(expected_private, wallet.listdescriptors(True))
 
         self.log.info('Test list private descriptors with watch-only wallet')
         node.createwallet(wallet_name='watch-only', descriptors=True, disable_private_keys=True)
@@ -102,7 +102,7 @@ class ListDescriptorsTest(BitcoinTestFramework):
             'desc': descsum_create('wpkh(' + xpub_acc + ')'),
             'timestamp': 1296688602,
         }])
-        assert_raises_rpc_error(-4, 'Can\'t get descriptor string', watch_only_wallet.listdescriptors, True)
+        # assert_raises_rpc_error(-4, 'Can\'t get descriptor string', watch_only_wallet.listdescriptors, True)
 
         self.log.info('Test non-active non-range combo descriptor')
         node.createwallet(wallet_name='w4', blank=True, descriptors=True)

'desc': descsum_create('wpkh(' + xpub_acc + ')'),
'timestamp': 1296688602,
}])
assert_raises_rpc_error(-4, 'Can\'t get descriptor string', watch_only_wallet.listdescriptors, True)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(pico-nit, only if you retouch)

Suggested change
assert_raises_rpc_error(-4, 'Can\'t get descriptor string', watch_only_wallet.listdescriptors, True)
assert_raises_rpc_error(-4, "Can't get descriptor string", watch_only_wallet.listdescriptors, True)

@@ -71,11 +71,39 @@ def run_test(self):
],
}
assert_equal(expected, wallet.listdescriptors())
assert_equal(expected, wallet.listdescriptors(False))
Copy link
Member

@jonatack jonatack Jul 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe use named args (as this argument is new and true/false isn't very descriptive), feel free to ignore

         assert_equal(expected, wallet.listdescriptors())
-        assert_equal(expected, wallet.listdescriptors(False))
+        assert_equal(expected, wallet.listdescriptors(private=False))
 
         self.log.info('Test list private descriptors')
         expected_private = {
@@ -84,16 +84,16 @@ class ListDescriptorsTest(BitcoinTestFramework):
                  'next': 0},
             ],
         }
-        assert_equal(expected_private, wallet.listdescriptors(True))
+        assert_equal(expected_private, wallet.listdescriptors(private=True))
 
         self.log.info("Test listdescriptors with encrypted wallet")
         wallet.encryptwallet("pass")
         assert_equal(expected, wallet.listdescriptors())
 
         self.log.info('Test list private descriptors with encrypted wallet')
-        assert_raises_rpc_error(-13, 'Please enter the wallet passphrase with walletpassphrase first.', wallet.listdescriptors, True)
+        assert_raises_rpc_error(-13, 'Please enter the wallet passphrase with walletpassphrase first.', wallet.listdescriptors, private=True)
         wallet.walletpassphrase(passphrase="pass", timeout=1000000)
-        assert_equal(expected_private, wallet.listdescriptors(True))
+        assert_equal(expected_private, wallet.listdescriptors(private=True))
 
         self.log.info('Test list private descriptors with watch-only wallet')
         node.createwallet(wallet_name='watch-only', descriptors=True, disable_private_keys=True)
@@ -102,7 +102,7 @@ class ListDescriptorsTest(BitcoinTestFramework):
             'desc': descsum_create('wpkh(' + xpub_acc + ')'),
             'timestamp': 1296688602,
         }])
-        assert_raises_rpc_error(-4, 'Can\'t get descriptor string', watch_only_wallet.listdescriptors, True)
+        assert_raises_rpc_error(-4, 'Can\'t get descriptor string', watch_only_wallet.listdescriptors, private=True)
 
         self.log.info('Test non-active non-range combo descriptor')
         node.createwallet(wallet_name='w4', blank=True, descriptors=True)

@achow101
Copy link
Member

re-ACK bb822a7

@S3RK
Copy link
Contributor Author

S3RK commented Aug 3, 2021

@prayank23 @Rspigler do you want to check the latest version?

@ghost
Copy link

ghost commented Aug 3, 2021

Sure. Will try today. I had two questions, maybe @practicalswift can help me:

  1. Does it make sense to try SQL queries in fuzzing descriptor wallet related RPCs? I tried but could not find anything interesting: https://link.medium.com/iQBbB5Tbqib
  2. Are emojis a part of strings that are used in fuzzing Bitcoin Core? https://github.com/danielmiessler/SecLists/blob/master/Fuzzing/big-list-of-naughty-strings.txt (few emojis in this list)

Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tACK bb822a7

Tested everything mentioned in PR description and below things:

W1: Descriptor wallet with private keys
W2: Blank Descriptor wallet
W3: Descriptor wallet with private keys disabled
W4: Another Descriptor wallet with private keys

  1. Export descriptors from wallet W1 and import in same wallet
  2. Export descriptors from wallet W1 and import in Wallet W2 and W4
  3. Export descriptors from wallet W1 and import in Wallet W3

Everything looks good with listdescriptors, maybe we can improve one thing in importdescriptors:

When exporting-importing in same wallet it does not import duplicates however response is same as 2.

{
  "success": true
}

@ghost
Copy link

ghost commented Aug 3, 2021

In case of watch-only wallet without private keys there will be following output:
error code: -4
error message:
Can't get descriptor string.

I don't see this warning for watch only wallet. It returns wallet name and blank array for descriptors.

@Rspigler
Copy link
Contributor

Rspigler commented Aug 4, 2021

tACK bb822a7

This would be very helpful for metal or paper backups

listdescriptors:

{
"wallet_name": "Test_ListDescriptors",
"descriptors": [
{
"desc": "pkh([db98369e/44'/0'/0']xpub6DFFBCcFzoG7cfKDPRX39zd3ZBydFnM26VmDEVDLryPU6u6coYZKkBpe7DKPa8V1RjAvvdx4UebiXqSirkevEpsdByAbS1GjF1WGofMWcNc/0/)#6j60t7ur",
"timestamp": 1628016073,
"active": true,
"internal": false,
"range": [
0,
999
],
"next": 0
},
{
"desc": "sh(wpkh([db98369e/49'/0'/0']xpub6DG6d9WNZi2EwnkKm8AZTRTijvff5kiMUPFyQN7aDiNJ5Hu7Nea4zeMVi7g8st1PxNKVq6gidk5vPiV2uMyu8QgmCa92A9XaXfZ7ZfCMrZP/0/
))#luvn8xr5",
"timestamp": 1628016073,
"active": true,
"internal": false,
"range": [
0,
999
],
"next": 0
},
{
"desc": "wpkh([db98369e/84'/0'/0']xpub6CPCUJH2M2VkskpSijywKte99eQ1WXZnVtTBEbWUCWKzHB36ddWSwmmg4jL4M2Nk8PRK3XS4YbTrKk5sCx1iCEgjKsZU2Vp5xQvgThhNQFH/0/)#g22fa57g",
"timestamp": 1628016074,
"active": true,
"internal": false,
"range": [
0,
999
],
"next": 0
},
{
"desc": "pkh([db98369e/44'/0'/0']xpub6DFFBCcFzoG7cfKDPRX39zd3ZBydFnM26VmDEVDLryPU6u6coYZKkBpe7DKPa8V1RjAvvdx4UebiXqSirkevEpsdByAbS1GjF1WGofMWcNc/1/
)#txlwktvm",
"timestamp": 1628016074,
"active": true,
"internal": true,
"range": [
0,
999
],
"next": 0
},
{
"desc": "sh(wpkh([db98369e/49'/0'/0']xpub6DG6d9WNZi2EwnkKm8AZTRTijvff5kiMUPFyQN7aDiNJ5Hu7Nea4zeMVi7g8st1PxNKVq6gidk5vPiV2uMyu8QgmCa92A9XaXfZ7ZfCMrZP/1/))#2az9lekt",
"timestamp": 1628016074,
"active": true,
"internal": true,
"range": [
0,
999
],
"next": 0
},
{
"desc": "wpkh([db98369e/84'/0'/0']xpub6CPCUJH2M2VkskpSijywKte99eQ1WXZnVtTBEbWUCWKzHB36ddWSwmmg4jL4M2Nk8PRK3XS4YbTrKk5sCx1iCEgjKsZU2Vp5xQvgThhNQFH/1/
)#e70gqpws",
"timestamp": 1628016075,
"active": true,
"internal": true,
"range": [
0,
999
],
"next": 0
}
]
}

listdescriptors true:

{
"wallet_name": "Test_ListDescriptors",
"descriptors": [
{
"desc": "pkh(xprv9s21ZrQH143K2CBvvodeKZrxF474uqw846zfy9Q4hWEtkuNZoSRdUrtPjntix8JXUh6M5VXPFEw7z8w7jvMkqZEP2D5qVoyfvmMC2SVfHc5/44'/0'/0'/0/)#3ln4pscr",
"timestamp": 1628016073,
"active": true,
"internal": false,
"range": [
0,
999
],
"next": 0
},
{
"desc": "sh(wpkh(xprv9s21ZrQH143K2CBvvodeKZrxF474uqw846zfy9Q4hWEtkuNZoSRdUrtPjntix8JXUh6M5VXPFEw7z8w7jvMkqZEP2D5qVoyfvmMC2SVfHc5/49'/0'/0'/0/
))#ulkh8xrl",
"timestamp": 1628016073,
"active": true,
"internal": false,
"range": [
0,
999
],
"next": 0
},
{
"desc": "wpkh(xprv9s21ZrQH143K2CBvvodeKZrxF474uqw846zfy9Q4hWEtkuNZoSRdUrtPjntix8JXUh6M5VXPFEw7z8w7jvMkqZEP2D5qVoyfvmMC2SVfHc5/84'/0'/0'/0/)#8mevkeep",
"timestamp": 1628016074,
"active": true,
"internal": false,
"range": [
0,
999
],
"next": 0
},
{
"desc": "pkh(xprv9s21ZrQH143K2CBvvodeKZrxF474uqw846zfy9Q4hWEtkuNZoSRdUrtPjntix8JXUh6M5VXPFEw7z8w7jvMkqZEP2D5qVoyfvmMC2SVfHc5/44'/0'/0'/1/
)#qtk5u9gm",
"timestamp": 1628016074,
"active": true,
"internal": true,
"range": [
0,
999
],
"next": 0
},
{
"desc": "sh(wpkh(xprv9s21ZrQH143K2CBvvodeKZrxF474uqw846zfy9Q4hWEtkuNZoSRdUrtPjntix8JXUh6M5VXPFEw7z8w7jvMkqZEP2D5qVoyfvmMC2SVfHc5/49'/0'/0'/1/))#6u7jutgt",
"timestamp": 1628016074,
"active": true,
"internal": true,
"range": [
0,
999
],
"next": 0
},
{
"desc": "wpkh(xprv9s21ZrQH143K2CBvvodeKZrxF474uqw846zfy9Q4hWEtkuNZoSRdUrtPjntix8JXUh6M5VXPFEw7z8w7jvMkqZEP2D5qVoyfvmMC2SVfHc5/84'/0'/0'/1/
)#k0udtvfe",
"timestamp": 1628016075,
"active": true,
"internal": true,
"range": [
0,
999
],
"next": 0
}
]
}

Copy link
Contributor

@meshcollider meshcollider left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review ACK bb822a7

@meshcollider meshcollider merged commit 8fa03c4 into bitcoin:master Aug 9, 2021
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Aug 10, 2021
…iptors

bb822a7 wallet, rpc: add listdescriptors private option (S3RK)

Pull request description:

  Rationale: make it possible to backup your wallet with `listdescriptors` command

  * The default behaviour is still to show public version
  * For private version only the root xprv is returned

  Example use-case:
  ```
  > bitcoin-cli -regtest -named createwallet wallet_name=old descriptors=true
  > bitcoin-cli -regtest -rpcwallet=old listdescriptors true | jq '.descriptors' > descriptors.txt

  > bitcoin-cli -regtest -named createwallet wallet_name=new descriptors=true blank=true
  > bitcoin-cli -regtest -rpcwallet=new importdescriptors "$(cat descriptors.txt)"
  ```

  In case of watch-only wallet without private keys there will be following output:
  ```
  error code: -4
  error message:
  Can't get descriptor string.
  ```

ACKs for top commit:
  achow101:
    re-ACK bb822a7
  Rspigler:
    tACK bb822a7
  jonatack:
    ACK bb822a7 per `git diff 2854ddc bb822a7`
  prayank23:
    tACK bitcoin@bb822a7
  meshcollider:
    Code review ACK bb822a7

Tree-SHA512: f6dddc72a74e5667071ccd77f8dce578382e8e29e7ed6a0834ac2e114a6d3918b59c2f194f4079b3259e13d9ba3b4f405619940c3ecb7a1a0344615aed47c43d
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants