Skip to content

Conversation

meshcollider
Copy link
Contributor

@meshcollider meshcollider commented Oct 16, 2018

Based on #14454 #14565, last two commits only are for review.

Best reviewed with ?w=1

Allows a descriptor to be imported into the wallet using importmulti RPC. Start and end of range can be specified for ranged descriptors. The descriptor is implicitly converted to old structures on import.

Also adds a simple test of a P2SH-P2WPKH address being imported as a descriptor. More tests to come, as well as release notes.

if (::IsMine(*pwallet, raw_pubkey_script) == ISMINE_SPENDABLE) {
throw JSONRPCError(RPC_WALLET_ERROR, "The wallet already contains the private key for this public key");
}
if (!pwallet->AddWatchOnly(raw_pubkey_script, timestamp)) {
Copy link
Member

@sipa sipa Oct 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the same reason as pointed out in the other PR, you can't do this; you're going to mark payments to individual multisig pubkeys as incoming payments.

You'll need a way to restrict this to P2PK, P2WPKH, and P2SH/P2WSH wrapped versions of those.

EDIT: I realize that when it's about private keys, the same effect applies too, and there it can't be avoided. Perhaps this stuff is just to scary, and we should wait until there's a way to actually specify what to treat as ours explicitly...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restricting it from multisig(to avoid being tricked as you mention) would make this even more confusing to a user.

Unfortunate.

Copy link
Contributor Author

@meshcollider meshcollider Oct 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mentioned on IRC, but what if we just never import public keys at all, and only either A) add the scriptPubKey as watch only or B) import the private key. If we have the private key then IMO it's less scary, because it's still "ours" and we can access the funds

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right; the same concern doesn't exist for private keys as you're obviously able to spend those coins anyway.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would not importing the pubkeys affect the wallet's ability to construct PSBT inputs?

Copy link
Member

@sipa sipa Oct 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only for things that have PKH/WPKH construction in them (GetPubKey is needed for those, which finds a pubkey based on pubkeyhash).

@meshcollider To be clear, we do have to import the pubkey for those; otherwise the result is not solvable.

Copy link
Contributor

@promag promag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some comments while reading the code. Will test.


success = true;
}
if (data.exists("scriptPubKey")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (data.exists("scriptPubKey") && data.exists("descriptor")) {
    // throw error because these should be exclusive?
}

and add test.

throw JSONRPCError(RPC_INVALID_ADDRESS_OR_KEY, "Descriptor is invalid");
}
if (!parsed_desc->IsRange() && data.exists("range")) {
throw JSONRPCError(RPC_INVALID_PARAMETER, "Range should not be specified for an un-ranged descriptor");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing test.

if (!parsed_desc->IsRange() && data.exists("range")) {
throw JSONRPCError(RPC_INVALID_PARAMETER, "Range should not be specified for an un-ranged descriptor");
} else if (parsed_desc->IsRange() && !data.exists("range")) {
throw JSONRPCError(RPC_INVALID_PARAMETER, "Descriptor is ranged, please specify the range");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing test.

} else if (data.exists("descriptor")) {
ProcessImportDesc(pwallet, data, timestamp);
} else {
throw JSONRPCError(RPC_INVALID_PARAMETER, "Either a descriptor or scriptPubKey must be provided.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing test.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I said more tests were coming 😅

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still coming?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@meshcollider
Copy link
Contributor Author

After discussion with sipa, closing for now, will come back to this after #14454 is merged

@achow101
Copy link
Member

achow101 commented Nov 1, 2018

Since #14454 has been merged, this can be reopened?

@sipa
Copy link
Member

sipa commented Nov 1, 2018

I believe a simpler approach is possible on top of #14565.

@meshcollider
Copy link
Contributor Author

Yep I'll reopen this when rebased on 14565

@meshcollider meshcollider reopened this Nov 3, 2018
@meshcollider
Copy link
Contributor Author

meshcollider commented Nov 3, 2018

Rebased on #14565

Still planning on adding more tests + release notes, please don't nitpick the lack of tests yet

Copy link
Member

@Sjors Sjors left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK, lightly tested 6894bf5 and it's able to import a test watch-only wallet I used previously.

I didn't use the watchonly while importing a descriptor with an xpub, should that be allowed? Or should watchonly argument not be allowed when using a descriptor?

Much smaller code change than I would have thought, nice!

It would be nice if range had a default of [0, DEFAULT_KEYPOOL_SIZE] or even using the -keypool argument.

Can you add a usage example with a descriptor?

When I cherry-pick #14477 on top of this and inspect a used address with getaddressinfo, the origin info in the descriptor seems both wrong and incomplete: "desc": "wpkh([224885f8]026...)", where 224885f8 is not the master fingerprint I used, and derivation info is missing. There might be some reusable stuff in #14021.

If I use watchonly: true, then getaddressinfo doesn't show a descriptor and it says solvable: false, which seems wrong (the latter also happens without cherry-pick).

@meshcollider
Copy link
Contributor Author

@Sjors thanks for the feedback :)

If I use watchonly: true, then getaddressinfo doesn't show a descriptor and it says solvable: false, which seems wrong (the latter also happens without cherry-pick).

It should only watch for the scriptPubKey and import no other information, watch-only is a different requirement than being solvable without private keys.

the origin info in the descriptor seems both wrong and incomplete: "desc": "wpkh([224885f8]026...)", where 224885f8 is not the master fingerprint I used, and derivation info is missing.

You're right, good point. I'll take a look at andrews PR

I didn't use the watchonly while importing a descriptor with an xpub, should that be allowed? Or should watchonly argument not be allowed when using a descriptor?

I think it should be allowed, because of the above reason. Using an xpub without watch only allows it to be solvable without being spendable

@sipa
Copy link
Member

sipa commented Nov 5, 2018

@meshcollider @Sjors I was imagining that "watchonly" would be implicit when using descriptors (addr() and raw() would be watchonly; anything else would result in a solvable result).

The reason for "watchonly" is so that users need to be explicit about the fact they don't want solvability (generally, you should always want solvability, but if you can't, you can tell importmulti that it's fine without).

@Sjors
Copy link
Member

Sjors commented Nov 6, 2018

@sipa I like your suggestion. In that case we should disallow usage of the watchonly param when combined with a descriptor.

Regarding getting origin information from the descriptors, @achow101 just rebased #14021 on top of this PR. It's a non trivial change. Perhaps for this PR it's best to not store origin information. Just make sure that if you do walletcreatefundedpsbt with bip32 flag set to true, then the result doesn't crash decodepsbt.

@meshcollider
Copy link
Contributor Author

I don't see the problem if you want to add a ranged descriptor as watch only to import all the scriptPubKeys but not retain any more info than that?

@Sjors
Copy link
Member

Sjors commented Nov 6, 2018

IIUC you would indicate that in the descriptor by wrapping the result in addr(...). Mandating that removes ambiguity from how a descriptor ends up in a wallet.

@meshcollider
Copy link
Contributor Author

meshcollider commented Nov 6, 2018

@Sjors It was my understanding that addr(...) cannot contain another type of descriptor or key (e.g. a ranged BIP32 key), only the base58/bech32 encoded address formats. If I'm wrong then I'm happy to change this PR, @sipa could weigh in here

@instagibbs
Copy link
Member

xpub byte prefix mismatch results in very generic error; would be nice to give something more meaningful

@DrahtBot
Copy link
Contributor

DrahtBot commented Nov 8, 2018

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #15032 (Nit fixes for PR 14565 by MeshCollider)
  • #14918 (RPCHelpMan: Check default values are given at compile-time by MarcoFalke)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@instagibbs
Copy link
Member

instagibbs commented Nov 9, 2018

Can you rebase this on most recent #14565 ? It has a few logic fixes. discussing what I think are bugs

In addition, you can not ignore the watch_only field for descriptor import. watch_only needs to be set to true if the descriptor deals with public keys only: https://github.com/bitcoin/bitcoin/pull/14565/files#diff-522490d83dce5375d423b23886e4125eR1019 (this code is from the refreshed dependent PR)

@Sjors
Copy link
Member

Sjors commented Dec 12, 2018

Now might be a good to time to rebase it, because testing this might reveal anything we've missed in the overhaul PR #14565.

@meshcollider
Copy link
Contributor Author

I've rebased this but haven't tested, it was a bit of a messy rebase and still a couple of things to address such as adding the warnings to the descriptor import function, plus more tests. I'll hopefully finish those things off in the next couple of days too.

@meshcollider
Copy link
Contributor Author

I've rebased and squashed as well as added some more tests for descriptor imports and fixed the public key issue now #15263 has gone in

Copy link
Member

@Sjors Sjors left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK b985e9c

FlatSigningProvider out_keys;

// Expand all descriptors to get public keys and scripts.
// TODO: get private keys from descriptors too
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: add (TODO to add) expandRange method to Descriptor instance

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sjors this TODO is addressed in #15024 :)

@achow101
Copy link
Member

achow101 commented Feb 5, 2019

utACK b985e9c

Copy link
Contributor

@ryanofsky ryanofsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK b985e9c. Changes since last review: rebase, more tests, changes to comments, documentation, and error checking.

// Check if this private key corresponds to a public key from the descriptor
if (!pubkey_map.count(id)) {
throw JSONRPCError(RPC_INVALID_ADDRESS_OR_KEY, "Unused private key provided");
warnings.push_back("Ignoring irrelevant private key.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #14491 (comment)

These comments don't appear to be resolved, but I think that's ok.

// This does not take into account threshold multisigs which could be spendable without all keys.
// Thus, threshold multisigs without all keys will be considered not spendable here, even if they are,
// perhaps triggering a false warning message. This is consistent with the current wallet IsMine check.
bool spendable = std::all_of(pubkey_map.begin(), pubkey_map.end(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "[wallet] Allow descriptor imports with importmulti" (9f48053)

It would be really nice to move this duplicated watchonly checking code out of ProcessLegacy and ProcessImportDescriptor either up into ProcessImport or down to into a common helper function. The watchonly checks are weird enough to deal with once, much less twice in two functions with slightly different variable names and comments.

// This does not take into account threshold multisigs which could be spendable without all keys
bool spendable = std::all_of(pubkey_map.begin(), pubkey_map.end(), [&](const std::pair<CKeyID, CPubKey>& used_key){ return privkey_map.count(used_key.first) > 0; });
if (!watch_only && !spendable) {
warnings.push_back("Some private keys are missing, outputs will be considered watchonly. If this is intentional, specify the watchonly flag.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #14491 (comment)

s/specify the watchonly flag/specify the watchonly flag to suppress this warning/

Note: implementing this suggestion would make the warning text vary between ProcessImportLegacy and ProcessImportDescriptor.

// Generate the script and destination for the scriptPubKey provided
CScript script;
CTxDestination dest;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "[wallet] Allow descriptor imports with importmulti" (9f48053)

Note: Changes here more are directly related to previous commit "[wallet] Refactor ProcessImport()" (a1b25e1) than this one.

import_data.import_scripts.emplace(x.second);
}

std::copy(out_keys.pubkeys.begin(), out_keys.pubkeys.end(), std::inserter(pubkey_map, pubkey_map.end()));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #14491 (comment)

For future reference, this was fixed by removing out.pubkeys.emplace in #15263.

I think it would be helpful to have a comment here noting that this imports the public keys which occur inside P2PKH and P2WPKH descriptors, but not those inside multisig descriptors. It would also be great to include or link to sipa's rationale for this in #14491 (comment)

@laanwj laanwj merged commit b985e9c into bitcoin:master Feb 7, 2019
laanwj added a commit that referenced this pull request Feb 7, 2019
b985e9c Add release notes for importmulti descriptor support (MeshCollider)
fbb5e93 Add test for importing via descriptor (MeshCollider)
9f48053 [wallet] Allow descriptor imports with importmulti (MeshCollider)
d2b381c [wallet] Refactor ProcessImport() to call ProcessImportLegacy() (John Newbery)
4cac0dd [wallet] Add ProcessImportLegacy() (John Newbery)
a1b25e1 [wallet] Refactor ProcessImport() (John Newbery)

Pull request description:

  ~~Based on #14454 #14565, last two commits only are for review.~~

  Best reviewed with `?w=1`

  Allows a descriptor to be imported into the wallet using `importmulti` RPC. Start and end of range can be specified for ranged descriptors. The descriptor is implicitly converted to old structures on import.

  Also adds a simple test of a P2SH-P2WPKH address being imported as a descriptor. More tests to come, as well as release notes.

Tree-SHA512: 160eb6fd574c4ae5b70e0109f7e5ccc95d9309138603408a1114ceb3c558065409c0d7afb66926bc8e1743c365a3b300c5f944ff18b2451acc0514fbeca1f2b3
@meshcollider meshcollider deleted the 201810_importmulti_desc_2 branch February 8, 2019 03:12
laanwj added a commit that referenced this pull request Jun 7, 2019
53b7de6 Add test for dumping the private key imported from descriptor (MeshCollider)
2857bc4 Extend importmulti descriptor tests (MeshCollider)
81a884b Import private keys from descriptor with importmulti if provided (MeshCollider)
a4d1bd1 Add private key derivation functions to descriptors (MeshCollider)

Pull request description:

  ~This is based on #14491, review the last 3 commits only.~

  Currently, descriptors have an Expand() function which returns public keys and scripts for a specific index of a ranged descriptor. But the private key for a specific index is not given. This allows private keys for specific indices to be derived. This also allows those keys to be imported through the `importmulti` RPC rather than having to provide them separately.

ACKs for commit 53b7de:
  achow101:
    ACK 53b7de6

Tree-SHA512: c060bc01358a1adc76d3d470fefc2bdd39c837027f452e9bc4bd2e726097e1ece4af9d5627efd942a5f8819271e15ba54f010b169b50a9435a1f0f40fd1cebf3
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jun 7, 2019
…escriptor

53b7de6 Add test for dumping the private key imported from descriptor (MeshCollider)
2857bc4 Extend importmulti descriptor tests (MeshCollider)
81a884b Import private keys from descriptor with importmulti if provided (MeshCollider)
a4d1bd1 Add private key derivation functions to descriptors (MeshCollider)

Pull request description:

  ~This is based on bitcoin#14491, review the last 3 commits only.~

  Currently, descriptors have an Expand() function which returns public keys and scripts for a specific index of a ranged descriptor. But the private key for a specific index is not given. This allows private keys for specific indices to be derived. This also allows those keys to be imported through the `importmulti` RPC rather than having to provide them separately.

ACKs for commit 53b7de:
  achow101:
    ACK 53b7de6

Tree-SHA512: c060bc01358a1adc76d3d470fefc2bdd39c837027f452e9bc4bd2e726097e1ece4af9d5627efd942a5f8819271e15ba54f010b169b50a9435a1f0f40fd1cebf3
deadalnix pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request May 19, 2020
Summary:
This commit is move-only and doesn't make any functional changes. It
simply moves code around within ProcessImport() in preparation for
refactors in the next commits.

This is a partial backport of Core [[bitcoin/bitcoin#14491 | PR14491]] : bitcoin/bitcoin@a1b25e1

Some loop variable names have been changed to make sure we don't have shaddowing going on.

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D6141
deadalnix pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request May 21, 2020
Summary:
This commit adds a ProcessImportLegacy() function which
currently does nothing. It also unindents a block of
code for a future move-only change.

Reviewer hint: review with -w to ignore whitespace changes.

This is a partial backport of Core [[bitcoin/bitcoin#14491 | PR14491]] : bitcoin/bitcoin@4cac0dd

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Subscribers: Fabien

Differential Revision: https://reviews.bitcoinabc.org/D6189
deadalnix pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request May 21, 2020
Summary:
This is almost entirely a move-only commit.

Reviewer hint: use --color-moved=zebra for review.

This is a partial backport of Core [[bitcoin/bitcoin#14491 | PR14491]] : bitcoin/bitcoin@d2b381c

Depends on D6189

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D6190
deadalnix pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request May 21, 2020
Summary:
This is a partial backport of Core [[bitcoin/bitcoin#14491 | PR14491]] : bitcoin/bitcoin@9f48053

Depends on D6190

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Subscribers: Fabien

Differential Revision: https://reviews.bitcoinabc.org/D6191
deadalnix pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request May 21, 2020
Summary:
This is a partial backport of Core [[bitcoin/bitcoin#14491 | PR14491]] : bitcoin/bitcoin@fbb5e93

Depends on D6187 and D6191

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D6192
deadalnix pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request May 21, 2020
Summary:
This is a partial backport of Core [[bitcoin/bitcoin#14491 | PR14491]] : bitcoin/bitcoin@b985e9c

Depends on D6192

Test Plan: read

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D6193
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Nov 12, 2020
Summary: This wasn't ported from [[bitcoin/bitcoin#14491 | PR14491]] because it used segwit, but this can be done without segwit.

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Subscribers: majcosta, PiRK

Differential Revision: https://reviews.bitcoinabc.org/D8359
kwvg added a commit to kwvg/dash that referenced this pull request Oct 12, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Oct 21, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Oct 25, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Oct 25, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Oct 26, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Oct 26, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Oct 28, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Oct 28, 2021
pravblockc pushed a commit to pravblockc/dash that referenced this pull request Nov 18, 2021
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Dec 16, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.