-
Notifications
You must be signed in to change notification settings - Fork 37.7k
wallet: Remove IsMine from migration code #30328
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/30328. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
744613e
to
574b840
Compare
574b840
to
d5d994c
Compare
Ready for review now that #26596 has been merged. |
src/wallet/scriptpubkeyman.cpp
Outdated
case TxoutType::MULTISIG: | ||
{ | ||
if (ctx == ScriptContext::P2WSH) { | ||
std::vector<valtype> keys(sols.begin() + 1, sols.begin() + sols.size() - 1); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In 905e22b "wallet: Remove IsMine from migration": Just a question, but could it be a vector of CPubKey
then use IsCompressed
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
HaveKeys
uses std::vector<valtype>
🚧 At least one of the CI tasks failed. HintsMake sure to run all tests locally, according to the documentation. The failure may happen due to a number of reasons, for example:
Leave a comment here, if you need help tracking down a confusing failure. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK, left two code-deduplication nits below
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚧 At least one of the CI tasks failed. HintsMake sure to run all tests locally, according to the documentation. The failure may happen due to a number of reasons, for example:
Leave a comment here, if you need help tracking down a confusing failure. |
d5d994c
to
b6c2aad
Compare
b6c2aad
to
705d9eb
Compare
#30328 (review) still applies to 705d9eb instead of d5d994c. (Clarified linked comment a bit). |
34751ae
to
ed5608a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Must admit that i'm not versed enough in the subtle specifics with regard to validity and spendability of scripts, to be sure all of this is correct. But where i've checked, the behavior matches the IsMine-based implementation. Overall changes LGTM.
src/wallet/scriptpubkeyman.cpp
Outdated
// to "spks", we can simply check whether this P2PK or P2PKH script is in "spk" to determine spendability. | ||
case TxoutType::PUBKEY: | ||
case TxoutType::PUBKEYHASH: | ||
if (spks.count(script) > 0) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doing both inserting into and reading from spks here makes it somewhat harder to be confident that the behavior is independent from the specific order of processing mapScripts, but the comment above helps.
fea8925
to
348e5aa
Compare
test/functional/wallet_migration.py
Outdated
wallet.unloadwallet() | ||
|
||
def test_disallowed_p2wsh(self): | ||
self.log.info("Test that P2WSH output scripts with invalid witnessScripts are not migrated") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is it important to test that an invalid script that we can import for historical reasons is not migrated? Are we testing that these artifacts do not cause the migration to fail rather than them not being migrated?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because invalid scripts can exist in the legacy wallet, but can cause issues (crashes, loading failure, etc.) in both migration and descriptor wallets. So we need to test that migration doesn't fail, and that those scripts do not end up in the descriptor wallet.
src/wallet/scriptpubkeyman.cpp
Outdated
// Bare multisigs are never considered spendable | ||
case TxoutType::MULTISIG: | ||
{ | ||
Assert(solutions.size() >= 2); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Errr, can a multisig be 0 of 0? If so, wouldn’t that only have a solution size of 1?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Solver returns TxoutType::NONSTANDARD
for 0-of-n, so this code cannot be reached in that case.
src/wallet/scriptpubkeyman.cpp
Outdated
if (!HaveKeys(keys, *this)) { | ||
break; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At this point, the witness program, witness script, or redeemscript is in our mapScript, and we might have enough keys for the quorum but just not all of the keys.
Why does this require all the keys to be present in the wallet rather than just the threshold?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To maintain strict compatibility with LegacyScriptPubKeyMan::IsMine()
. I don't think we should change this behavior in this PR.
348e5aa
to
e09407c
Compare
e09407c
to
76f1d5c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, the PR is in a much clearer state now with all the newly introduced comments and the P2WSH scripts loop removed.
test/functional/wallet_migration.py
Outdated
comp_eckey = ECKey() | ||
comp_eckey.generate(compressed=True) | ||
comp_pubkey = comp_eckey.get_pubkey().get_bytes().hex() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in commit 883030c: could use the generate_keypair
helper both here and for the uncompressed variant below
comp_eckey = ECKey() | |
comp_eckey.generate(compressed=True) | |
comp_pubkey = comp_eckey.get_pubkey().get_bytes().hex() | |
comp_eckey_wif, comp_pubkey = generate_keypair(compressed=True, wif=True) |
then the ECKey
and byte_to_base58
imports are not needed anymore
case TxoutType::MULTISIG: | ||
{ | ||
if (ctx == ScriptContext::P2WSH) { | ||
std::vector<std::vector<unsigned char>> keys(solutions.begin() + 1, solutions.begin() + solutions.size() - 1); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in commit 19dc8a3: same as in the MULTISIG Solver case
below (in the mapScripts loop), could also add an assertion about the solutions size here (or maybe even introduce some get_multisig_keys_from_solutions
lambda to further deduplicate)
std::vector<std::vector<unsigned char>> keys(solutions.begin() + 1, solutions.begin() + solutions.size() - 1); | |
Assert(solutions.size() >= 2); | |
std::vector<std::vector<unsigned char>> keys(solutions.begin() + 1, solutions.begin() + solutions.size() - 1); |
src/wallet/scriptpubkeyman.cpp
Outdated
// P2WSH-Multisig output scripts are spendable if the P2WSH output script is also in mapScripts, | ||
// and all keys are compressed | ||
CScript ms_wsh = GetScriptForDestination(WitnessV0ScriptHash(script)); | ||
if (HaveCScript(CScriptID(ms_wsh)) && all_keys_compressed(keys)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: could use is_valid_script
here to not repeat the logic (as also done in the PUBKEY
/PUBKEYHASH
cases above):
if (HaveCScript(CScriptID(ms_wsh)) && all_keys_compressed(keys)) { | |
if (HaveCScript(CScriptID(ms_wsh)) && is_valid_script(script, ScriptContext::P2WSH)) { |
As IsMine will be removed, the relevant components of IsMine are inlined into the migration functions.
This reverts commit b231f4d.
76f1d5c
to
d25c0fb
Compare
If the `releases` directory exists, but still only a subset of the necessary previous release binaries are available, the test fails by throwing an exception (sometimes leading to follow-up exceptions like "AssertionError: [node 0] Error: no RPC connection") and printing out a stack trace, which can be confusing and at a first glance suggests that the node crashed or some alike. Improve this by checking and printing out *all* of the missing release binaries and skip the test in this case. Noticed while testing bitcoin#30328. Can be tested by e.g. $ ./test/get_previous_releases -b $ rm -rf ./releases/v28.0/ $ ./build/test/functional/wallet_migration.py
Very concepty comment (after IRL discussion with @achow101 and @furszy) Since this code is effectively introducing an analogue of the existing I would suggest a fuzz test that constructs a legacy SPKM (populated with some keys, redeemscripts, watchscripts), then generates the set of scriptPubKeys to watch using the migration code, checks that all the scripts in that set are IsMine() according to the old code, and then also generate a number of other scriptPubKeys, and test that IsMine() is true for them if and only if they're in the set. A complication is that IsMine may return true for things we cannot actually migrate, but I believe that's fixable by in the test ignoring IsMine() output for anything that cannot be InferDescriptor'd to a valid/roundtrippable descriptor. I'm happy to help brainstorming how such a fuzz could work. |
Following our discussion yesterday on additional edge cases, I have concluded that this approach is no longer viable as it has a high likelihood of missing further edge cases. I have opened #31495 which uses the suggested alternative approach of filtering the upper bound of scripts with |
…issing If the `releases` directory exists, but still only a subset of the necessary previous release binaries are available, the test fails by throwing an exception (sometimes leading to follow-up exceptions like "AssertionError: [node 0] Error: no RPC connection") and printing out a stack trace, which can be confusing and at a first glance suggests that the node crashed or some alike. Improve this by checking and printing out *all* of the missing release binaries and failing with an explicit error in this case. Also add an info on how to download previous releases binaries. Noticed while testing bitcoin#30328. Can be tested by e.g. $ ./test/get_previous_releases.py -b $ rm -rf ./releases/v28.0/ $ ./build/test/functional/wallet_migration.py
…issing If the `releases` directory exists, but still only a subset of the necessary previous release binaries are available, the test fails by throwing an exception (sometimes leading to follow-up exceptions like "AssertionError: [node 0] Error: no RPC connection") and printing out a stack trace, which can be confusing and at a first glance suggests that the node crashed or some alike. Improve this by checking and printing out *all* of the missing release binaries and failing with an explicit error in this case. Also add an info on how to download previous releases binaries. Noticed while testing bitcoin#30328. Can be tested by e.g. $ ./test/get_previous_releases.py -b $ rm -rf ./releases/v28.0/ $ ./build/test/functional/wallet_migration.py
…issing If the `releases` directory exists, but still only a subset of the necessary previous release binaries are available, the test fails by throwing an exception (sometimes leading to follow-up exceptions like "AssertionError: [node 0] Error: no RPC connection") and printing out a stack trace, which can be confusing and at a first glance suggests that the node crashed or some alike. Improve this by checking and printing out *all* of the missing release binaries and failing with an explicit error in this case. Also add an info on how to download previous releases binaries. Noticed while testing bitcoin#30328. Can be tested by e.g. $ ./test/get_previous_releases.py -b $ rm -rf ./releases/v28.0/ $ ./build/test/functional/wallet_migration.py
… binaries is missing 1ea7e45 test: raise explicit error if any of the needed release binaries is missing (Sebastian Falbesoner) Pull request description: If the `releases` directory exists, but still only a subset of the necessary previous release binaries are available, the test fails by throwing an exception (sometimes leading to follow-up exceptions like `AssertionError: [node 0] Error: no RPC connection`) and printing out a stack trace, which can be confusing and at a first glance suggests that the node crashed or some alike. Improve this by checking and printing out *all* of the missing release binaries and failing with an explicit error in this case. Also add an info on how to download previous releases binaries. Noticed while testing #30328. Can be tested by e.g. ``` $ rm -rf ./releases $ ./test/get_previous_releases.py -b $ rm -rf ./releases/v28.0/ $ ./build/test/functional/wallet_migration.py ``` master: <details> <summary>Long test fail output</summary> ``` ... 2024-12-10T18:48:54.067000Z TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 590, in start_nodes node.start(extra_args[i], *args, **kwargs) File "/home/thestack/bitcoin/test/functional/test_framework/test_node.py", line 257, in start self.process = subprocess.Popen(self.args + extra_args, env=subp_env, stdout=stdout, stderr=stderr, cwd=cwd, **kwargs) File "/usr/lib/python3.10/subprocess.py", line 971, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: '/home/thestack/bitcoin/releases/v28.0/bin/bitcoind' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 131, in main self.setup() File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 315, in setup self.setup_network() File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 409, in setup_network self.setup_nodes() File "/home/thestack/bitcoin/./build/test/functional/wallet_migration.py", line 54, in setup_nodes self.start_nodes() File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 595, in start_nodes self.stop_nodes() File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 610, in stop_nodes node.stop_node(wait=wait, wait_until_stopped=False) File "/home/thestack/bitcoin/test/functional/test_framework/test_node.py", line 396, in stop_node self.stop(wait=wait) File "/home/thestack/bitcoin/test/functional/test_framework/test_node.py", line 215, in __getattr__ assert self.rpc_connected and self.rpc is not None, self._node_msg("Error: no RPC connection") AssertionError: [node 0] Error: no RPC connection 2024-12-10T18:48:54.118000Z TestFramework (INFO): Stopping nodes Traceback (most recent call last): File "/home/thestack/bitcoin/./build/test/functional/wallet_migration.py", line 1097, in <module> WalletMigrationTest(__file__).main() File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 159, in main exit_code = self.shutdown() File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 331, in shutdown self.stop_nodes() File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 610, in stop_nodes node.stop_node(wait=wait, wait_until_stopped=False) File "/home/thestack/bitcoin/test/functional/test_framework/test_node.py", line 396, in stop_node self.stop(wait=wait) File "/home/thestack/bitcoin/test/functional/test_framework/test_node.py", line 215, in __getattr__ assert self.rpc_connected and self.rpc is not None, self._node_msg("Error: no RPC connection") AssertionError: [node 0] Error: no RPC connection [node 0] Cleaning up leftover process ... ``` </details> PR: ``` ... 2025-01-01T20:26:27.999000Z TestFramework (INFO): PRNG seed is: 4570383538468804512 2025-01-01T20:26:28.000000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_lz66_zcu 2025-01-01T20:26:28.003000Z TestFramework (ERROR): Binary not found: /home/thestack/bitcoin/releases/v28.0/bin/bitcoind 2025-01-01T20:26:28.003000Z TestFramework (ERROR): Binary not found: /home/thestack/bitcoin/releases/v28.0/bin/bitcoin-cli 2025-01-01T20:26:28.003000Z TestFramework (INFO): Previous releases binaries can be downloaded via `test/get_previous_releases.py -b`. 2025-01-01T20:26:28.003000Z TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 131, in main self.setup() File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 315, in setup self.setup_network() File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 409, in setup_network self.setup_nodes() File "/home/thestack/bitcoin/./build/test/functional/wallet_migration.py", line 50, in setup_nodes self.add_nodes(self.num_nodes, versions=[ File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 537, in add_nodes raise AssertionError("At least one release binary is missing") AssertionError: At least one release binary is missing 2025-01-01T20:26:28.061000Z TestFramework (INFO): Stopping nodes ... ``` ACKs for top commit: fjahr: re-ACK 1ea7e45 kevkevinpal: ACK [1ea7e45](1ea7e45) maflcko: lgtm ACK 1ea7e45 achow101: ACK 1ea7e45 pablomartin4btc: tACK 1ea7e45 Tree-SHA512: b621c3ce044ca8fc8715a4f4b1f96a8592a470c319a64444cced9ba692d315cfd4885a066679bf377b19136fa3530d9cff6f18894a45aa9c716d39b12719baa0
…to cover edge cases af76664 test: Test migration of a solvable script with no privkeys (Ava Chow) 17f01b0 test: Test migration of taproot output scripts (Ava Chow) 1eb9a2a test: Test migration of miniscript in legacy wallets (Ava Chow) e8c3efc wallet migration: Determine Solvables with CanProvide (Ava Chow) fa1b7cd migration: Skip descriptors which do not parse (Ava Chow) 440ea1a legacy spkm: use IsMine() to extract watched output scripts (Ava Chow) b777e84 legacy spkm: Move CanProvide to LegacyDataSPKM (Ava Chow) b1ab927 tests: Test migration of additional P2WSH scripts (Ava Chow) c39b3cf test: Extra verification that migratewallet migrates (Ava Chow) Pull request description: The legacy wallet `IsMine()` is essentially a black box that would tell us whether the wallet is watching an output script. In order to migrate legacy wallets to descriptor wallets, we need to be able to compute all of the output scripts that a legacy wallet would watch. The original approach for this was to understand `IsMine()` and write a function which would be its inverse. This was partially done in the original migration code, and attempted to be completed in #30328. However, further analysis of `IsMine()` has continued to reveal additional edge cases which make writing an inverse function increasingly difficult to verify correctness. This PR instead changes migration to utilize `IsMine()` to produce the output scripts by first computing a superset of all of the output scripts that `IsMine()` would watch and testing each script against `IsMine()` to filter for the ones that actually are watched. The superset is constructed by computing all possible output scripts for the keys and scripts in the wallet - for keys, every key could be a P2PK, P2PKH, P2WPKH, and P2SH-P2WPKH; for scripts, every script could be an output script, the redeemScript of a P2SH, the witnessScript of a P2WSH, and the witnessScript of a P2SH-P2WSH. Additionally, the legacy wallet can contain scripts that are redeemScripts and witnessScripts, while not watching for any output script utilizing that script. These are known as solvable scripts and are migrated to a separate "solvables" wallet. The previous approach to identifying these solvables was similar to identifying output scripts - finding known solvable conditions and computing the scripts. However, this also can miss scripts, so the solvables are now identified in a manner similar to the output scripts but using the function `CanProvide()`. Using the same superset as before, all output scripts which are `ISMINE_NO` are put through `CanProvide()` which will perform a dummy signing and then a key lookup to determine whether the legacy wallet could provide any solving data for the output script. The scripts that pass will have their descriptors inferred and the script included in the solvables wallet. The main downside of this approach is that `IsMine()` and `CanProvide()` can no longer be deleted. They will need to be refactored to be migration only code instead in #28710. Lastly, I've added 2 test cases for the edge cases that prompted this change of approach. In particular, miniscript witnessScripts and `rawtr()` output scripts are solvable and signable in a legacy wallet, although never `ISMINE_SPENDABLE`. ACKs for top commit: sipa: Code review ACK af76664; I did not review the tests in detail. brunoerg: code review ACK af76664 rkrux: ACK af76664 Tree-SHA512: 7f58a90de6f38fe9801fb6c2a520627072c8d66358652ad0872ff59deb678a82664b99babcfd874288bebcb1487d099a77821f03ae063c2b4cbf2d316e77d141
The legacy wallet
IsMine
code will be removed with the legacy wallet, but is still partially needed for migration. Instead of usingIsMine
directly in migration, equivalent checks are performed by migration.Builds on #26596