-
Notifications
You must be signed in to change notification settings - Fork 37.7k
Miniscript integration follow-ups #24860
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
4.0K /tmp/cirrus-ci-build/depends/sdk-sources/
4.0K /tmp/cirrus-ci-build/releases/x86_64-pc-linux-gnu
184 fuzz target(s) found: addition_overflow addr_info_deserialize address_deserialize_v1_notime address_deserialize_v1_withtime address_deserialize_v2 addrman addrman_deserialize addrman_serdeser asmap asmap_direct autofile banman base_encode_decode bech32 block block_deserialize block_file_info_deserialize block_filter_deserialize block_header block_header_and_short_txids_deserialize blockfilter blockheader_deserialize blocklocator_deserialize blockmerkleroot blocktransactions_deserialize blocktransactionsrequest_deserialize blockundo_deserialize bloom_filter bloomfilter_deserialize buffered_file chain checkqueue coins_deserialize coins_view coinselection connman crypto crypto_aes256 crypto_aes256cbc crypto_chacha20 crypto_chacha20_poly1305_aead crypto_common crypto_diff_fuzz_chacha20 crypto_hkdf_hmac_sha256_l32 crypto_poly1305 cuckoocache data_stream_addr_man decode_tx descriptor_parse diskblockindex_deserialize eval_script fee_rate fee_rate_deserialize fees flat_file_pos_deserialize flatfile float golomb_rice hex http_request i2p integer inv_deserialize key key_io key_origin_info_deserialize kitchen_sink load_external_block_file locale merkle_block_deserialize merkleblock message messageheader_deserialize miniscript_script miniscript_string minisketch muhash multiplication_overflow net net_permissions netaddr_deserialize netaddress netbase_dns_lookup node_eviction out_point_deserialize p2p_transport_serialization parse_hd_keypath parse_iso8601 parse_numbers parse_script parse_univalue partial_merkle_tree_deserialize partially_signed_transaction_deserialize policy_estimator policy_estimator_io pow prefilled_transaction_deserialize prevector primitives_transaction process_message process_message_addr process_message_addrv2 process_message_block process_message_blocktxn process_message_cfcheckpt process_message_cfheaders process_message_cfilter process_message_cmpctblock process_message_feefilter process_message_filteradd process_message_filterclear process_message_filterload process_message_getaddr process_message_getblocks process_message_getblocktxn process_message_getcfcheckpt process_message_getcfheaders process_message_getcfilters process_message_getdata process_message_getheaders process_message_headers process_message_inv process_message_mempool process_message_merkleblock process_message_notfound process_message_ping process_message_pong process_message_sendaddrv2 process_message_sendcmpct process_message_sendheaders process_message_tx process_message_verack process_message_version process_message_wtxidrelay process_messages protocol psbt psbt_input_deserialize psbt_output_deserialize pub_key_deserialize random rbf rolling_bloom_filter rpc script script_bitcoin_consensus script_descriptor_cache script_deserialize script_flags script_format script_interpreter script_ops script_sigcache script_sign scriptnum_ops secp256k1_ec_seckey_import_export_der secp256k1_ecdsa_signature_parse_der_lax service_deserialize signature_checker signet snapshotmetadata_deserialize socks5 span spanparsing str_printf string system timedata torcontrol transaction tx_in tx_in_deserialize tx_out tx_pool tx_pool_standard txoutcompressor_deserialize txrequest txundo_deserialize uint160_deserialize uint256_deserialize utxo_snapshot validation_load_mempool versionbits wallet_notifications
184 of 184 detected fuzz target(s) selected: addition_overflow addr_info_deserialize address_deserialize_v1_notime address_deserialize_v1_withtime address_deserialize_v2 addrman addrman_deserialize addrman_serdeser asmap asmap_direct autofile banman base_encode_decode bech32 block block_deserialize block_file_info_deserialize block_filter_deserialize block_header block_header_and_short_txids_deserialize blockfilter blockheader_deserialize blocklocator_deserialize blockmerkleroot blocktransactions_deserialize blocktransactionsrequest_deserialize blockundo_deserialize bloom_filter bloomfilter_deserialize buffered_file chain checkqueue coins_deserialize coins_view coinselection connman crypto crypto_aes256 crypto_aes256cbc crypto_chacha20 crypto_chacha20_poly1305_aead crypto_common crypto_diff_fuzz_chacha20 crypto_hkdf_hmac_sha256_l32 crypto_poly1305 cuckoocache data_stream_addr_man decode_tx descriptor_parse diskblockindex_deserialize eval_script fee_rate fee_rate_deserialize fees flat_file_pos_deserialize flatfile float golomb_rice hex http_request i2p integer inv_deserialize key key_io key_origin_info_deserialize kitchen_sink load_external_block_file locale merkle_block_deserialize merkleblock message messageheader_deserialize miniscript_script miniscript_string minisketch muhash multiplication_overflow net net_permissions netaddr_deserialize netaddress netbase_dns_lookup node_eviction out_point_deserialize p2p_transport_serialization parse_hd_keypath parse_iso8601 parse_numbers parse_script parse_univalue partial_merkle_tree_deserialize partially_signed_transaction_deserialize policy_estimator policy_estimator_io pow prefilled_transaction_deserialize prevector primitives_transaction process_message process_message_addr process_message_addrv2 process_message_block process_message_blocktxn process_message_cfcheckpt process_message_cfheaders process_message_cfilter process_message_cmpctblock process_message_feefilter process_message_filteradd process_message_filterclear process_message_filterload process_message_getaddr process_message_getblocks process_message_getblocktxn process_message_getcfcheckpt process_message_getcfheaders process_message_getcfilters process_message_getdata process_message_getheaders process_message_headers process_message_inv process_message_mempool process_message_merkleblock process_message_notfound process_message_ping process_message_pong process_message_sendaddrv2 process_message_sendcmpct process_message_sendheaders process_message_tx process_message_verack process_message_version process_message_wtxidrelay process_messages protocol psbt psbt_input_deserialize psbt_output_deserialize pub_key_deserialize random rbf rolling_bloom_filter rpc script script_bitcoin_consensus script_descriptor_cache script_deserialize script_flags script_format script_interpreter script_ops script_sigcache script_sign scriptnum_ops secp256k1_ec_seckey_import_export_der secp256k1_ecdsa_signature_parse_der_lax service_deserialize signature_checker signet snapshotmetadata_deserialize socks5 span spanparsing str_printf string system timedata torcontrol transaction tx_in tx_in_deserialize tx_out tx_pool tx_pool_standard txoutcompressor_deserialize txrequest txundo_deserialize uint160_deserialize uint256_deserialize utxo_snapshot validation_load_mempool versionbits wallet_notifications
Fuzzing harnesses lacking a corpus: coinselection miniscript_script miniscript_string wallet_notifications
Please consider adding a fuzz corpus at https://github.com/bitcoin-core/qa-assets
Run address_deserialize_v2 with args ['/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/fuzz/fuzz', '-runs=1', '/tmp/cirrus-ci-build/ci/scratch/qa-assets/fuzz_seed_corpus/address_deserialize_v2']fuzz: key.cpp:392: void ECC_Start(): Assertion `secp256k1_context_sign == nullptr' failed.
fuzz: key.cpp:392: void ECC_Start(): Assertion `secp256k1_context_sign == nullptr' failed.
Target "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/fuzz/fuzz -runs=1 /tmp/cirrus-ci-build/ci/scratch/qa-assets/fuzz_seed_corpus/address_deserialize_v2" failed with exit code -6 |
Did some very light code review, it looks sane. I didn't look at the fuzz changes. Disallowing duplicate keys seems reasonable; I would not at all be surprised if people generate miniscript by hand or even from Sipa's website, and then paste whatever keys they need into the result. E.g. if you just need a timelock fallback, which you can't do with current descriptors, you probably won't go through the trouble of finding and installing a compiler, figuring out how to put xpub's into it, etc. Nit: can you make b9a36d3 a scripted diff? |
9071194
to
ad634a5
Compare
Thanks for the notification about the fuzzer failure, i fixed it. See also discussion about it in sipa/miniscript#106. Regarding the scripted diff, sure. I'll push it soon. |
ad634a5
to
e9828f2
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK, with some opinionable comments!
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
cc @sanket1729 |
e9828f2
to
426848e
Compare
426848e
to
22b2171
Compare
Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
The 'Fragment' type was previously named 'Nodetype'. For clarity, name the variables the same. -BEGIN VERIFY SCRIPT- sed -i 's/nodetype/fragment/g' src/script/miniscript.* -END VERIFY SCRIPT- Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
This helps to have finer-grained descriptor parsing errors.
This makes IsSane clearer. It is useful to differentiate between 'potential non-malleable satisfactions are valid' and 'such satisfactions exist' for testing. Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
Parse also key hashes using the Key type. Make this target the first of the 4 Miniscript fuzz targets in a single `miniscript` file. Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
22b2171
to
038f367
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Few questions.
src/script/miniscript.h
Outdated
bool ValidSatisfactions() const { return IsValid() && CheckOpsLimit() && CheckStackSize(); } | ||
|
||
//! Whether the apparent policy of this node matches its script semantics. | ||
bool IsSane() const { return ValidSatisfactions() && IsNonMalleable() && CheckTimeLocksMix(); } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Technically, all timelocks are malleable if there is no signature. If there is no signature, anyone can malleate nLockTime/nSequence to any value perhaps not matching the script semantics?
Why is the NeedsSignature check only on top-level? and not a general condition for sanity?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are right, n:older(12)
shouldn't be called "sane". But to call it "sane" one should use IsSaneTopLevel()
.
I think that the check for a fragment's sanity should be context-less, and the requirement for signature depends on the context. To take back the example from above n:older(12)
shouldn't be called sane on its own, but it should in the context of and_b(n:older(12),s:pk(A))
. So it should only be called insane if not 's' and is the top level fragment. That's what IsSaneTopLevel()
is.
It's not purely theoretical, for instance we use this in #24148 to find the deepest insane sub of an insane Miniscript: https://github.com/bitcoin/bitcoin/pull/24148/files#diff-a55760aaec4bce216663f5ebf65823516347356a8320d30459427149f7bbc2c5R824-R831. If the IsSane()
check is context-dependent it gets messy.
Now, maybe there is a point for the need for a better naming of these methods.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I think this is right.
IsSane() being false for n:older(12)
would be strange, because while indeed it would be bad to use that as a standalone script, it is perfectly reasonable (and even expected) to occur as a subexpression inside other scripts.
What about swapping the naming around?
- IsSane() -> IsSaneSubexpression()
- IsSaneTopLevel() -> IsSane()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
038f367
to
225d73c
Compare
I think i addressed all comments here. It's ready for another round of review. :) I had previously made the rename a scripted diff as asked by @Sjors. In the last push i:
|
src/script/miniscript.h
Outdated
bool ValidSatisfactions() const { return IsValid() && CheckOpsLimit() && CheckStackSize(); } | ||
|
||
//! Whether the apparent policy of this node matches its script semantics. | ||
bool IsSane() const { return ValidSatisfactions() && IsNonMalleable() && CheckTimeLocksMix(); } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I think this is right.
IsSane() being false for n:older(12)
would be strange, because while indeed it would be bad to use that as a standalone script, it is perfectly reasonable (and even expected) to occur as a subexpression inside other scripts.
What about swapping the naming around?
- IsSane() -> IsSaneSubexpression()
- IsSaneTopLevel() -> IsSane()
225d73c
to
6111d2c
Compare
6111d2c
to
59010da
Compare
Added an explicit requirement on the context to provide a |
59010da
to
f552056
Compare
utACK f552056 (with the caveat that a lot of it is my own code). With or without addressing #24860 (comment) |
As stated on the website, duplicate keys make it hard to reason about malleability as a single signature may unlock multiple paths. We use a custom KeyCompare function instead of operator< to be explicit about the requirement.
Suggested-by: Vincenzo Palazzo
f552056
to
f3a50c9
Compare
utACK f3a50c9 (with the caveat that a lot of it is my own code) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
code review ACK f3a50c9. Did not review the fuzz tests.
f3a50c9 miniscript: rename IsSane and IsSaneSubexpression to prevent misuse (Antoine Poinsot) c5fe516 miniscript: nit: don't return after assert(false) (Antoine Poinsot) 7bbaca9 miniscript: explicit the threshold size computation in multi() (Antoine Poinsot) 8323e42 miniscript: add an OpCode typedef for readability (Antoine Poinsot) 7a549c6 miniscript: mark nodes with duplicate keys as insane (Antoine Poinsot) 8c0f8bf fuzz: add a Miniscript target for string representation roundtripping (Antoine Poinsot) be34d50 fuzz: rename and improve the Miniscript Script roundtrip target (Antoine Poinsot) 7eb70f0 miniscript: tiny doc fixups (Antoine Poinsot) 5cea85f miniscript: split ValidSatisfactions from IsSane (Antoine Poinsot) a0f064d miniscript: introduce a CheckTimeLocksMix helper (Antoine Poinsot) ed45ee3 miniscript: use optional instead of bool/outarg (Antoine Poinsot) 1ab8d89 miniscript: make equality operator non-recursive (Antoine Poinsot) 5922c66 scripted-diff: miniscript: rename 'nodetype' variables to 'fragment' (Antoine Poinsot) c5f65db miniscript: remove a workaround for a GCC 4.8 bug (Antoine Poinsot) Pull request description: The Miniscript repository and the Miniscript integration PR here have been a moving target for the past months, and some final cleanups were done there that were not included here. I initially intended to add some small followup commits to bitcoin#24148 but i think there are enough of them to be worth a followup PR on its own. Some parts of the code did not change since it was initially written in 2019, and the code could use some modernization. (Use std::optional instead of out args, remove old compiler workarounds). We refactored the helpers to be more meaningful, and also did some renaming. A new fuzz target was also added and both were merged in a single file. 2 more will be added in bitcoin#24149 that will be contained in this file too. The only behaviour change in this PR is to rule out Miniscript with duplicate keys from sane Miniscripts. In a P2WSH context, signatures can be rebounded (Miniscript does not use CODESEPARATOR) and it's reasonable to assume that reusing keys across the Script drops the malleability guarantees. It was previously assumed such Miniscript would never exist in the first place since a compiler should never create them. We finally agreed that if one were to exist (say, written by hand or from a buggy compiler) it would be very confusing if an imported Miniscript descriptor (after bitcoin#24148) with duplicate keys was deemed sane (ie, "safe to use") by Bitcoin Core. We now check for duplicate keys in the constructor. This is (still) joint work with Pieter Wuille. (Actually he entirely authored the cleanups and code modernization.) ACKs for top commit: sipa: utACK f3a50c9 (with the caveat that a lot of it is my own code) sanket1729: code review ACK f3a50c9. Did not review the fuzz tests. Tree-SHA512: c043325e4936fe25e8ece4266b46119e000c6745f88cea530fed1edf01c80f03ee6f9edc83b6e9d42ca01688d184bad16bfd967c5bb8037744e726993adf3deb
Renamed the fuzz target input folder to preserve the inputs: bitcoin-core/qa-assets@3366f7b |
I've been populating corpora for the two targets here. I'll PR them to qa-assets soon.
------- Original Message -------
Le mercredi 8 juin 2022 à 8:09 PM, MacroFake ***@***.***> a écrit :
… Renamed the fuzz target input folder to preserve the inputs: ***@***.***(bitcoin-core/qa-assets@3366f7b)
—
Reply to this email directly, [view it on GitHub](#24860 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AFLK3F2HMLUCLFEVIE3XPQLVODOUHANCNFSM5TP67B6Q).
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
{ | ||
FuzzedDataProvider provider(buffer.data(), buffer.size()); | ||
auto str = provider.ConsumeRemainingBytesAsString(); | ||
auto parsed = miniscript::FromString(str, PARSER_CTX); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like this may take quite a while. For example:
clusterfuzz-testcase-miniscript_string-5717444439703552.bin.txt
takes 10 seconds
With the flame graph:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah calling it in the constructor was not exactly bright since we'll re-do the same computation as we pile more nodes in the parsers. On the other hand it's hard not to do so and to keep the const
ness of the NodeRef
. Here are different approaches i tried:
- Cache the duplicate key result only once we call it https://github.com/darosior/bitcoin/tree/miniscript_cache_dup_key_check. Needs to drop the constness.
- Conditionally skip the check in the constructor, in order to only do it for the top level node https://github.com/darosior/bitcoin/tree/miniscript_skip_dup_key_check. Doesn't drop the constness but not elegant..
- Don't cache it at all https://github.com/darosior/bitcoin/tree/miniscript_no_dup_key_cache
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@darosior What about this idea:
- Make the current
miniscript::Node
constructors private, and make them not compute theduplicate_key
value. - Add a forwarding public constructor which invokes the private ones, and also computes the
duplicate_key
value (so now any publicly constructed Node object will have duplicate_key set, but it's possible to bypass this computation by using the private ones, if you have access to them). FromScript
andFromString
are madefriend
, and thus can use the private constructors.- Make the
duplicate_key
variable mutable, soContainsDuplicateKey
can modify it (and is changed so it sets the value for all children of the called node too).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds much better than all my approaches. I'll implement and PR this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm about to PR the fix, doing final testing at the moment.
…prevent memory blowup when fuzzing e8cc2e4 Make miniscript string parsing account for exact script size as bound (Pieter Wuille) 4cb8f9a Permit delaying duplicate key check in miniscript::Node construction (Pieter Wuille) Pull request description: As reported in bitcoin/bitcoin#24860 (comment), the current code to construct a `miniscript::Node` could cause a blowup on large fuzzer inputs. This is because: 1. The duplicate key check is redundantly done at parsing time, since we will recursively create miniscript nodes and the constructor will unconditionally look for duplicate across this node's keys and all its sub-nodes'. 2. We don't put an upper bound on the size of the inputs to consider for parsing. To avoid wasteful computation, and prevent the blowup on some fuzzer inputs, limit the size of reasonable inputs and only perform the check for duplicate keys once when parsing. Regarding the duplicate key check bypass in the constructor we iterated on different approaches, and eventually settled on passing a dummy argument. Albeit less elegant, all other approaches required getting rid of `std::make_shared` and adding an allocation *per node created*. This PR contains code from Pieter Wuille (see commits). Fixes bitcoin/bitcoin#25824. ACKs for top commit: darosior: ACK e8cc2e4 -- it's my own PR but most of the code here was written by sipa. I've reviewed and tested it. sipa: ACK e8cc2e4 (for the few parts of the code that aren't mine) Tree-SHA512: c21de39b3eeb484393758629882fcf8694a9bd1b8f15ae22efcec1582efc9c2309c5a0c2d90f361dd8e233d704a07dcd5fb982f4a48a002c4d8789e1d78bb526
…memory blowup when fuzzing e8cc2e4 Make miniscript string parsing account for exact script size as bound (Pieter Wuille) 4cb8f9a Permit delaying duplicate key check in miniscript::Node construction (Pieter Wuille) Pull request description: As reported in bitcoin#24860 (comment), the current code to construct a `miniscript::Node` could cause a blowup on large fuzzer inputs. This is because: 1. The duplicate key check is redundantly done at parsing time, since we will recursively create miniscript nodes and the constructor will unconditionally look for duplicate across this node's keys and all its sub-nodes'. 2. We don't put an upper bound on the size of the inputs to consider for parsing. To avoid wasteful computation, and prevent the blowup on some fuzzer inputs, limit the size of reasonable inputs and only perform the check for duplicate keys once when parsing. Regarding the duplicate key check bypass in the constructor we iterated on different approaches, and eventually settled on passing a dummy argument. Albeit less elegant, all other approaches required getting rid of `std::make_shared` and adding an allocation *per node created*. This PR contains code from Pieter Wuille (see commits). Fixes bitcoin#25824. ACKs for top commit: darosior: ACK e8cc2e4 -- it's my own PR but most of the code here was written by sipa. I've reviewed and tested it. sipa: ACK e8cc2e4 (for the few parts of the code that aren't mine) Tree-SHA512: c21de39b3eeb484393758629882fcf8694a9bd1b8f15ae22efcec1582efc9c2309c5a0c2d90f361dd8e233d704a07dcd5fb982f4a48a002c4d8789e1d78bb526
The Miniscript repository and the Miniscript integration PR here have been a moving target for the past months, and some final cleanups were done there that were not included here. I initially intended to add some small followup commits to #24148 but i think there are enough of them to be worth a followup PR on its own.
Some parts of the code did not change since it was initially written in 2019, and the code could use some modernization. (Use std::optional instead of out args, remove old compiler workarounds).
We refactored the helpers to be more meaningful, and also did some renaming. A new fuzz target was also added and both were merged in a single file. 2 more will be added in #24149 that will be contained in this file too.
The only behaviour change in this PR is to rule out Miniscript with duplicate keys from sane Miniscripts. In a P2WSH context, signatures can be rebounded (Miniscript does not use CODESEPARATOR) and it's reasonable to assume that reusing keys across the Script drops the malleability guarantees.
It was previously assumed such Miniscript would never exist in the first place since a compiler should never create them. We finally agreed that if one were to exist (say, written by hand or from a buggy compiler) it would be very confusing if an imported Miniscript descriptor (after #24148) with duplicate keys was deemed sane (ie, "safe to use") by Bitcoin Core. We now check for duplicate keys in the constructor.
This is (still) joint work with Pieter Wuille. (Actually he entirely authored the cleanups and code modernization.)