Skip to content

Conversation

practicalswift
Copy link
Contributor

@practicalswift practicalswift commented Oct 4, 2019

Add fuzzing harnesses for DecodeRawPSBT(...), ParseHDKeypath(...), ParseScript(...), various number parsing functions and various JSON/univalue parsing functions.

Testing this PR
As usual the best way to test proposed fuzzing harnesses is to use test_fuzzing_harnesses.sh (#17000) to quickly verify that the relevant code regions are triggered, that the fuzzing throughput seems reasonable, etc.

test_fuzzing_harnesses.sh 'psbt|hd_keypath|numbers|parse_script|univalue' 10 runs all fuzzers matching the regexp and gives them ten seconds of runtime each.

$ CC=clang CXX=clang++ ./configure --enable-fuzz --with-sanitizers=address,fuzzer,undefined
$ make
$ contrib/devtools/test_fuzzing_harnesses.sh 'psbt|hd_keypath|numbers|parse_script|univalue' 10
Testing fuzzer parse_hd_keypath during 10 second(s)
A subset of reached functions:
        NEW_FUNC[0/2]: 0x55bc23a76940 in ParsePrechecks(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/util/strencodings.cpp:267
        NEW_FUNC[1/2]: 0x55bc23a77300 in ParseUInt32(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int*) src/util/strencodings.cpp:309
stat::number_of_executed_units: 34237
stat::average_exec_per_sec:     3112
stat::new_units_added:          113
stat::slowest_unit_time_sec:    0
stat::peak_rss_mb:              282
Number of unique code paths taken during fuzzing round: 30

Testing fuzzer parse_numbers during 10 second(s)
A subset of reached functions:
stat::number_of_executed_units: 31309
stat::average_exec_per_sec:     2846
stat::new_units_added:          688
stat::slowest_unit_time_sec:    0
stat::peak_rss_mb:              234
Number of unique code paths taken during fuzzing round: 149

Testing fuzzer parse_script during 10 second(s)
A subset of reached functions:
        NEW_FUNC[1/11]: 0x5636ff61ba00 in IsDigit(char) src/./util/strencodings.h:70
        NEW_FUNC[0/14]: 0x5636fe6c6280 in CScript::operator<<(opcodetype) src/./script/script.h:448
        NEW_FUNC[1/14]: 0x5636fe6e0290 in prevector<28u, unsigned char, unsigned int, int>::insert(prevector<28u, unsigned char, unsigned int, int>::iterator, unsigned char const&) src/./prevector.h:342
        NEW_FUNC[2/14]: 0x5636fe6e1040 in prevector<28u, unsigned char, unsigned int, int>::size() const src/./prevector.h:277
        NEW_FUNC[3/14]: 0x5636fe6e1250 in prevector<28u, unsigned char, unsigned int, int>::capacity() const src/./prevector.h:295
        NEW_FUNC[4/14]: 0x5636fe6e1cb0 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) src/./prevector.h:196
        NEW_FUNC[0/10]: 0x5636fe6c5650 in CScript::operator<<(std::vector<unsigned char, std::allocator<unsigned char> > const&) src/./script/script.h:462
        NEW_FUNC[2/10]: 0x5636fe6e0a20 in void prevector<28u, unsigned char, unsigned int, int>::insert<__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > > >(prevector<28u, unsigned char, unsigned int, int>::iterator, __gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<[32/1902]
char> > >, __gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > >) src/./prevector.h:368
        NEW_FUNC[5/10]: 0x5636fe6e2350 in void prevector<28u, unsigned char, unsigned int, int>::fill<__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > > >(unsigned char*, __gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > >, __gnu_cxx::__normal_iterator<unsign
ed char const*, std::vector<unsigned char, std::allocator<unsigned char> > >) src/./prevector.h:204
        NEW_FUNC[0/1]: 0x5636ff8e48b0 in IsHex(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/util/strencodings.cpp:61
        NEW_FUNC[0/2]: 0x5636fe6e1410 in prevector<28u, unsigned char, unsigned int, int>::change_capacity(unsigned int) src/./prevector.h:165
        NEW_FUNC[1/2]: 0x5636fe6e1f00 in prevector<28u, unsigned char, unsigned int, int>::indirect_ptr(int) src/./prevector.h:161
        NEW_FUNC[0/1]: 0x5636fe6e0580 in void prevector<28u, unsigned char, unsigned int, int>::insert<unsigned char*>(prevector<28u, unsigned char, unsigned int, int>::iterator, unsigned char*, unsigned char*) src/./prevector.h:368
        NEW_FUNC[0/3]: 0x5636fe85f0d0 in CScript::push_int64(long) src/./script/script.h:394
        NEW_FUNC[1/3]: 0x5636fe85f520 in prevector<28u, unsigned char, unsigned int, int>::push_back(unsigned char const&) src/./prevector.h:422
        NEW_FUNC[2/3]: 0x5636ff8ed730 in atoi64(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/util/strencodings.cpp:417
stat::number_of_executed_units: 8153
stat::average_exec_per_sec:     741
stat::new_units_added:          296
stat::slowest_unit_time_sec:    0
stat::peak_rss_mb:              237
Number of unique code paths taken during fuzzing round: 98

Testing fuzzer parse_univalue during 10 second(s)
A subset of reached functions:
        NEW_FUNC[0/19]: 0x560db8655950 in tinyformat::detail::formatImpl(std::ostream&, char const*, tinyformat::detail::FormatArg const*, int) src/./tinyformat.h:791
        NEW_FUNC[4/19]: 0x560db86582b0 in tinyformat::detail::printFormatStringLiteral(std::ostream&, char const*) src/./tinyformat.h:564
        NEW_FUNC[5/19]: 0x560db8658690 in tinyformat::detail::streamStateFromFormat(std::ostream&, bool&, int&, char const*, tinyformat::detail::FormatArg const*, int&, int) src/./tinyformat.h:601
        NEW_FUNC[6/19]: 0x560db865f090 in tinyformat::detail::FormatArg::format(std::ostream&, char const*, char const*, int) const src/./tinyformat.h:513
        NEW_FUNC[12/19]: 0x560db8661ba0 in void tinyformat::detail::FormatArg::formatImpl<int>(std::ostream&, char const*, char const*, int, void const*) src/./tinyformat.h:530
        NEW_FUNC[13/19]: 0x560db8661d90 in void tinyformat::formatValue<int>(std::ostream&, char const*, char const*, int, int const&) src/./tinyformat.h:317
        NEW_FUNC[14/19]: 0x560db875c8b0 in void tinyformat::detail::FormatArg::formatImpl<unsigned int>(std::ostream&, char const*, char const*, int, void const*) src/./tinyformat.h:530
        NEW_FUNC[15/19]: 0x560db875caa0 in void tinyformat::formatValue<unsigned int>(std::ostream&, char const*, char const*, int, unsigned int const&) src/./tinyformat.h:317
        NEW_FUNC[16/19]: 0x560db9473ef0 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > tinyformat::format<int, unsigned int>(char const*, int const&, unsigned int const&) src/./tinyformat.h:976
        NEW_FUNC[17/19]: 0x560db94749a0 in void tinyformat::format<int, unsigned int>(std::ostream&, char const*, int const&, unsigned int const&) src/./tinyformat.h:968
        NEW_FUNC[18/19]: 0x560db9474cf0 in tinyformat::detail::FormatListN<2>::FormatListN<int, unsigned int>(int const&, unsigned int const&) src/./tinyformat.h:885
stat::number_of_executed_units: 14089
stat::average_exec_per_sec:     1280
stat::new_units_added:          135
stat::slowest_unit_time_sec:    0
stat::peak_rss_mb:              356
Number of unique code paths taken during fuzzing round: 62

Testing fuzzer psbt_input_deserialize during 10 second(s)
A subset of reached functions:
        NEW_FUNC[0/46]: 0x557847ce3530 in prevector<28u, unsigned char, unsigned int, int>::~prevector() src/./prevector.h:456
        NEW_FUNC[3/46]: 0x557847cfdcf0 in prevector<28u, unsigned char, unsigned int, int>::size() const src/./prevector.h:277
        NEW_FUNC[4/46]: 0x557847cfe0c0 in prevector<28u, unsigned char, unsigned int, int>::change_capacity(unsigned int) src/./prevector.h:165
        NEW_FUNC[13/46]: 0x557847d3c890 in unsigned long ReadCompactSize<CDataStream>(CDataStream&) src/./serialize.h:290
        NEW_FUNC[14/46]: 0x557847d47b60 in prevector<28u, unsigned char, unsigned int, int>::resize(unsigned int) src/./prevector.h:311
        NEW_FUNC[16/46]: 0x557847d48800 in CTxOut::CTxOut() src/./primitives/transaction.h:140
        NEW_FUNC[17/46]: 0x557847d4b050 in CTxOut::SetNull() src/./primitives/transaction.h:155
        NEW_FUNC[18/46]: 0x557847d4b140 in CScript::clear() src/./script/script.h:563
        NEW_FUNC[19/46]: 0x557847d4ead0 in void Unserialize_impl<CDataStream, unsigned char, std::allocator<unsigned char> >(CDataStream&, std::vector<unsigned char, std::allocator<unsigned char> >&, unsigned char const&) src/./serialize.h:746
        NEW_FUNC[0/58]: 0x557847cfdf00 in prevector<28u, unsigned char, unsigned int, int>::capacity() const src/./prevector.h:295
        NEW_FUNC[1/58]: 0x557847cfe960 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) src/./prevector.h:196
        NEW_FUNC[2/58]: 0x557847cfebb0 in prevector<28u, unsigned char, unsigned int, int>::indirect_ptr(int) src/./prevector.h:161
        NEW_FUNC[3/58]: 0x557847d03990 in uint256::uint256() src/./uint256.h:123
        NEW_FUNC[0/3]: 0x557847d47430 in void CScript::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./script/script.h:418
        NEW_FUNC[1/3]: 0x557847d47730 in void Unserialize_impl<CDataStream, 28u, unsigned char>(CDataStream&, prevector<28u, unsigned char, unsigned int, int>&, unsigned char const&) src/./serialize.h:666
        NEW_FUNC[2/3]: 0x557847d60dd0 in CDataStream& CDataStream::operator>><CScript&>(CScript&) src/./streams.h:460
        NEW_FUNC[1/78]: 0x557847cffae0 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) const src/./prevector.h:197
        NEW_FUNC[2/78]: 0x557847cffd30 in prevector<28u, unsigned char, unsigned int, int>::indirect_ptr(int) const src/./prevector.h:162
        NEW_FUNC[0/1]: 0x557847d65f90 in OverrideStream<CDataStream>& OverrideStream<CDataStream>::operator>><unsigned char&>(unsigned char&) src/./streams.h:46
        NEW_FUNC[0/3]: 0x557847d470e0 in void SerReadWriteMany<CDataStream, CScript&>(CDataStream&, CSerActionUnserialize, CScript&) src/./serialize.h:989
        NEW_FUNC[1/3]: 0x557847d4ac50 in void CTxOut::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./primitives/transaction.h:149
        NEW_FUNC[2/3]: 0x557847d5f860 in void UnserializeFromVector<CDataStream, CTxOut>(CDataStream&, CTxOut&) src/./script/sign.h:90
        NEW_FUNC[0/1]: 0x557847d60840 in void UnserializeFromVector<CDataStream, int>(CDataStream&, int&) src/./script/sign.h:90
        NEW_FUNC[0/1]: 0x557847d41010 in CMutableTransaction::HasWitness() const src/./primitives/transaction.h:398
stat::number_of_executed_units: 13615
stat::average_exec_per_sec:     1237
stat::new_units_added:          357
stat::slowest_unit_time_sec:    0
stat::peak_rss_mb:              446
Number of unique code paths taken during fuzzing round: 152

Testing fuzzer psbt_output_deserialize during 10 second(s)
A subset of reached functions:
        NEW_FUNC[0/27]: 0x55c9347e5940 in prevector<28u, unsigned char, unsigned int, int>::~prevector() src/./prevector.h:456
        NEW_FUNC[5/27]: 0x55c93483eca0 in unsigned long ReadCompactSize<CDataStream>(CDataStream&) src/./serialize.h:290
        NEW_FUNC[6/27]: 0x55c934850ee0 in void Unserialize_impl<CDataStream, unsigned char, std::allocator<unsigned char> >(CDataStream&, std::vector<unsigned char, std::allocator<unsigned char> >&, unsigned char const&) src/./serialize.h:746
        NEW_FUNC[14/27]: 0x55c934858500 in PSBTOutput::PSBTOutput() src/./psbt.h:281
        NEW_FUNC[15/27]: 0x55c934858870 in CDataStream& CDataStream::operator>><PSBTOutput&>(PSBTOutput&) src/./streams.h:460
        NEW_FUNC[0/1]: 0x55c934800100 in prevector<28u, unsigned char, unsigned int, int>::size() const src/./prevector.h:277
        NEW_FUNC[0/4]: 0x55c934849840 in void CScript::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./script/script.h:418
        NEW_FUNC[1/4]: 0x55c934849b40 in void Unserialize_impl<CDataStream, 28u, unsigned char>(CDataStream&, prevector<28u, unsigned char, unsigned int, int>&, unsigned char const&) src/./serialize.h:666
        NEW_FUNC[2/4]: 0x55c934849f70 in prevector<28u, unsigned char, unsigned int, int>::resize(unsigned int) src/./prevector.h:311
        NEW_FUNC[3/4]: 0x55c93485dc60 in CDataStream& CDataStream::operator>><CScript&>(CScript&) src/./streams.h:460
        NEW_FUNC[0/3]: 0x55c934800310 in prevector<28u, unsigned char, unsigned int, int>::capacity() const src/./prevector.h:295
        NEW_FUNC[1/3]: 0x55c934800d70 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) src/./prevector.h:196
        NEW_FUNC[2/3]: 0x55c934849d40 in prevector<28u, unsigned char, unsigned int, int>::resize_uninitialized(unsigned int) src/./prevector.h:381
        NEW_FUNC[0/1]: 0x55c93485ddd0 in void DeserializeHDKeypaths<CDataStream>(CDataStream&, std::vector<unsigned char, std::allocator<unsigned char> > const&, std::map<CPubKey, KeyOriginInfo, std::less<CPubKey>, std::allocator<std::pair<CPubKey const, KeyOriginInfo> > >&) src/./script/sign.h:103
stat::number_of_executed_units: 19130
stat::average_exec_per_sec:     1739
stat::new_units_added:          195
stat::slowest_unit_time_sec:    0
stat::peak_rss_mb:              411
Number of unique code paths taken during fuzzing round: 64

Tested fuzz harnesses seem to work as expected.

@laanwj
Copy link
Member

laanwj commented Oct 4, 2019

Concept ACK, thanks for adding tests.

But please shorten the PR title a bit this is crazy 😄 it's meant to be a short summary

@fanquake fanquake added the Tests label Oct 4, 2019
@practicalswift practicalswift changed the title tests: Add fuzzing harnesses for DecodeRawPSBT(...), ParseHDKeypath(...), ParseScript(...), various number parsing functions and various JSON/univalue parsing functions tests: Add fuzzing harnesses for functions parsing scripts, numbers, JSON and HD keypaths (bip32) Oct 4, 2019
@DrahtBot
Copy link
Contributor

DrahtBot commented Oct 4, 2019

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #10443 (Add fee_est tool for debugging fee estimation code by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@practicalswift
Copy link
Contributor Author

Rebased! :)

@practicalswift
Copy link
Contributor Author

Rebased!

@practicalswift
Copy link
Contributor Author

Rebased! :)

@practicalswift
Copy link
Contributor Author

Rebased! :)

@practicalswift
Copy link
Contributor Author

Addressed @MarcoFalke's comment. Please re-review :)

maflcko pushed a commit that referenced this pull request Dec 11, 2019
…pts, numbers, JSON and HD keypaths (bip32)

a1308b7 tests: Add fuzzing harnesses for various JSON/univalue parsing functions (practicalswift)
e3d2bcf tests: Add fuzzing harnesses for various number parsing functions (practicalswift)
fb8c120 tests: Add ParseScript(...) (core_io) fuzzing harness (practicalswift)
074cb64 tests: Add ParseHDKeypath(...) (bip32) fuzzing harness (practicalswift)
0dc5907 tests: Add corpora suppression (FUZZERS_MISSING_CORPORA) for fuzzers missing in https://github.com/bitcoin-core/qa-assets/tree/master/fuzz_seed_corpus (practicalswift)

Pull request description:

  Add fuzzing harnesses for `DecodeRawPSBT(...)`, `ParseHDKeypath(...)`, `ParseScript(...)`, various number parsing functions and various JSON/univalue parsing functions.

  **Testing this PR**
  As usual the best way to test proposed fuzzing harnesses is to use `test_fuzzing_harnesses.sh` (#17000) to quickly verify that the relevant code regions are triggered, that the fuzzing throughput seems reasonable, etc.

  `test_fuzzing_harnesses.sh 'psbt|hd_keypath|numbers|parse_script|univalue' 10` runs all fuzzers matching the regexp and gives them ten seconds of runtime each.

  ```
  $ CC=clang CXX=clang++ ./configure --enable-fuzz --with-sanitizers=address,fuzzer,undefined
  $ make
  $ contrib/devtools/test_fuzzing_harnesses.sh 'psbt|hd_keypath|numbers|parse_script|univalue' 10
  Testing fuzzer parse_hd_keypath during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[0/2]: 0x55bc23a76940 in ParsePrechecks(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/util/strencodings.cpp:267
          NEW_FUNC[1/2]: 0x55bc23a77300 in ParseUInt32(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int*) src/util/strencodings.cpp:309
  stat::number_of_executed_units: 34237
  stat::average_exec_per_sec:     3112
  stat::new_units_added:          113
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              282
  Number of unique code paths taken during fuzzing round: 30

  Testing fuzzer parse_numbers during 10 second(s)
  A subset of reached functions:
  stat::number_of_executed_units: 31309
  stat::average_exec_per_sec:     2846
  stat::new_units_added:          688
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              234
  Number of unique code paths taken during fuzzing round: 149

  Testing fuzzer parse_script during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[1/11]: 0x5636ff61ba00 in IsDigit(char) src/./util/strencodings.h:70
          NEW_FUNC[0/14]: 0x5636fe6c6280 in CScript::operator<<(opcodetype) src/./script/script.h:448
          NEW_FUNC[1/14]: 0x5636fe6e0290 in prevector<28u, unsigned char, unsigned int, int>::insert(prevector<28u, unsigned char, unsigned int, int>::iterator, unsigned char const&) src/./prevector.h:342
          NEW_FUNC[2/14]: 0x5636fe6e1040 in prevector<28u, unsigned char, unsigned int, int>::size() const src/./prevector.h:277
          NEW_FUNC[3/14]: 0x5636fe6e1250 in prevector<28u, unsigned char, unsigned int, int>::capacity() const src/./prevector.h:295
          NEW_FUNC[4/14]: 0x5636fe6e1cb0 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) src/./prevector.h:196
          NEW_FUNC[0/10]: 0x5636fe6c5650 in CScript::operator<<(std::vector<unsigned char, std::allocator<unsigned char> > const&) src/./script/script.h:462
          NEW_FUNC[2/10]: 0x5636fe6e0a20 in void prevector<28u, unsigned char, unsigned int, int>::insert<__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > > >(prevector<28u, unsigned char, unsigned int, int>::iterator, __gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<[32/1902]
  char> > >, __gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > >) src/./prevector.h:368
          NEW_FUNC[5/10]: 0x5636fe6e2350 in void prevector<28u, unsigned char, unsigned int, int>::fill<__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > > >(unsigned char*, __gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > >, __gnu_cxx::__normal_iterator<unsign
  ed char const*, std::vector<unsigned char, std::allocator<unsigned char> > >) src/./prevector.h:204
          NEW_FUNC[0/1]: 0x5636ff8e48b0 in IsHex(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/util/strencodings.cpp:61
          NEW_FUNC[0/2]: 0x5636fe6e1410 in prevector<28u, unsigned char, unsigned int, int>::change_capacity(unsigned int) src/./prevector.h:165
          NEW_FUNC[1/2]: 0x5636fe6e1f00 in prevector<28u, unsigned char, unsigned int, int>::indirect_ptr(int) src/./prevector.h:161
          NEW_FUNC[0/1]: 0x5636fe6e0580 in void prevector<28u, unsigned char, unsigned int, int>::insert<unsigned char*>(prevector<28u, unsigned char, unsigned int, int>::iterator, unsigned char*, unsigned char*) src/./prevector.h:368
          NEW_FUNC[0/3]: 0x5636fe85f0d0 in CScript::push_int64(long) src/./script/script.h:394
          NEW_FUNC[1/3]: 0x5636fe85f520 in prevector<28u, unsigned char, unsigned int, int>::push_back(unsigned char const&) src/./prevector.h:422
          NEW_FUNC[2/3]: 0x5636ff8ed730 in atoi64(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/util/strencodings.cpp:417
  stat::number_of_executed_units: 8153
  stat::average_exec_per_sec:     741
  stat::new_units_added:          296
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              237
  Number of unique code paths taken during fuzzing round: 98

  Testing fuzzer parse_univalue during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[0/19]: 0x560db8655950 in tinyformat::detail::formatImpl(std::ostream&, char const*, tinyformat::detail::FormatArg const*, int) src/./tinyformat.h:791
          NEW_FUNC[4/19]: 0x560db86582b0 in tinyformat::detail::printFormatStringLiteral(std::ostream&, char const*) src/./tinyformat.h:564
          NEW_FUNC[5/19]: 0x560db8658690 in tinyformat::detail::streamStateFromFormat(std::ostream&, bool&, int&, char const*, tinyformat::detail::FormatArg const*, int&, int) src/./tinyformat.h:601
          NEW_FUNC[6/19]: 0x560db865f090 in tinyformat::detail::FormatArg::format(std::ostream&, char const*, char const*, int) const src/./tinyformat.h:513
          NEW_FUNC[12/19]: 0x560db8661ba0 in void tinyformat::detail::FormatArg::formatImpl<int>(std::ostream&, char const*, char const*, int, void const*) src/./tinyformat.h:530
          NEW_FUNC[13/19]: 0x560db8661d90 in void tinyformat::formatValue<int>(std::ostream&, char const*, char const*, int, int const&) src/./tinyformat.h:317
          NEW_FUNC[14/19]: 0x560db875c8b0 in void tinyformat::detail::FormatArg::formatImpl<unsigned int>(std::ostream&, char const*, char const*, int, void const*) src/./tinyformat.h:530
          NEW_FUNC[15/19]: 0x560db875caa0 in void tinyformat::formatValue<unsigned int>(std::ostream&, char const*, char const*, int, unsigned int const&) src/./tinyformat.h:317
          NEW_FUNC[16/19]: 0x560db9473ef0 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > tinyformat::format<int, unsigned int>(char const*, int const&, unsigned int const&) src/./tinyformat.h:976
          NEW_FUNC[17/19]: 0x560db94749a0 in void tinyformat::format<int, unsigned int>(std::ostream&, char const*, int const&, unsigned int const&) src/./tinyformat.h:968
          NEW_FUNC[18/19]: 0x560db9474cf0 in tinyformat::detail::FormatListN<2>::FormatListN<int, unsigned int>(int const&, unsigned int const&) src/./tinyformat.h:885
  stat::number_of_executed_units: 14089
  stat::average_exec_per_sec:     1280
  stat::new_units_added:          135
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              356
  Number of unique code paths taken during fuzzing round: 62

  Testing fuzzer psbt_input_deserialize during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[0/46]: 0x557847ce3530 in prevector<28u, unsigned char, unsigned int, int>::~prevector() src/./prevector.h:456
          NEW_FUNC[3/46]: 0x557847cfdcf0 in prevector<28u, unsigned char, unsigned int, int>::size() const src/./prevector.h:277
          NEW_FUNC[4/46]: 0x557847cfe0c0 in prevector<28u, unsigned char, unsigned int, int>::change_capacity(unsigned int) src/./prevector.h:165
          NEW_FUNC[13/46]: 0x557847d3c890 in unsigned long ReadCompactSize<CDataStream>(CDataStream&) src/./serialize.h:290
          NEW_FUNC[14/46]: 0x557847d47b60 in prevector<28u, unsigned char, unsigned int, int>::resize(unsigned int) src/./prevector.h:311
          NEW_FUNC[16/46]: 0x557847d48800 in CTxOut::CTxOut() src/./primitives/transaction.h:140
          NEW_FUNC[17/46]: 0x557847d4b050 in CTxOut::SetNull() src/./primitives/transaction.h:155
          NEW_FUNC[18/46]: 0x557847d4b140 in CScript::clear() src/./script/script.h:563
          NEW_FUNC[19/46]: 0x557847d4ead0 in void Unserialize_impl<CDataStream, unsigned char, std::allocator<unsigned char> >(CDataStream&, std::vector<unsigned char, std::allocator<unsigned char> >&, unsigned char const&) src/./serialize.h:746
          NEW_FUNC[0/58]: 0x557847cfdf00 in prevector<28u, unsigned char, unsigned int, int>::capacity() const src/./prevector.h:295
          NEW_FUNC[1/58]: 0x557847cfe960 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) src/./prevector.h:196
          NEW_FUNC[2/58]: 0x557847cfebb0 in prevector<28u, unsigned char, unsigned int, int>::indirect_ptr(int) src/./prevector.h:161
          NEW_FUNC[3/58]: 0x557847d03990 in uint256::uint256() src/./uint256.h:123
          NEW_FUNC[0/3]: 0x557847d47430 in void CScript::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./script/script.h:418
          NEW_FUNC[1/3]: 0x557847d47730 in void Unserialize_impl<CDataStream, 28u, unsigned char>(CDataStream&, prevector<28u, unsigned char, unsigned int, int>&, unsigned char const&) src/./serialize.h:666
          NEW_FUNC[2/3]: 0x557847d60dd0 in CDataStream& CDataStream::operator>><CScript&>(CScript&) src/./streams.h:460
          NEW_FUNC[1/78]: 0x557847cffae0 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) const src/./prevector.h:197
          NEW_FUNC[2/78]: 0x557847cffd30 in prevector<28u, unsigned char, unsigned int, int>::indirect_ptr(int) const src/./prevector.h:162
          NEW_FUNC[0/1]: 0x557847d65f90 in OverrideStream<CDataStream>& OverrideStream<CDataStream>::operator>><unsigned char&>(unsigned char&) src/./streams.h:46
          NEW_FUNC[0/3]: 0x557847d470e0 in void SerReadWriteMany<CDataStream, CScript&>(CDataStream&, CSerActionUnserialize, CScript&) src/./serialize.h:989
          NEW_FUNC[1/3]: 0x557847d4ac50 in void CTxOut::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./primitives/transaction.h:149
          NEW_FUNC[2/3]: 0x557847d5f860 in void UnserializeFromVector<CDataStream, CTxOut>(CDataStream&, CTxOut&) src/./script/sign.h:90
          NEW_FUNC[0/1]: 0x557847d60840 in void UnserializeFromVector<CDataStream, int>(CDataStream&, int&) src/./script/sign.h:90
          NEW_FUNC[0/1]: 0x557847d41010 in CMutableTransaction::HasWitness() const src/./primitives/transaction.h:398
  stat::number_of_executed_units: 13615
  stat::average_exec_per_sec:     1237
  stat::new_units_added:          357
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              446
  Number of unique code paths taken during fuzzing round: 152

  Testing fuzzer psbt_output_deserialize during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[0/27]: 0x55c9347e5940 in prevector<28u, unsigned char, unsigned int, int>::~prevector() src/./prevector.h:456
          NEW_FUNC[5/27]: 0x55c93483eca0 in unsigned long ReadCompactSize<CDataStream>(CDataStream&) src/./serialize.h:290
          NEW_FUNC[6/27]: 0x55c934850ee0 in void Unserialize_impl<CDataStream, unsigned char, std::allocator<unsigned char> >(CDataStream&, std::vector<unsigned char, std::allocator<unsigned char> >&, unsigned char const&) src/./serialize.h:746
          NEW_FUNC[14/27]: 0x55c934858500 in PSBTOutput::PSBTOutput() src/./psbt.h:281
          NEW_FUNC[15/27]: 0x55c934858870 in CDataStream& CDataStream::operator>><PSBTOutput&>(PSBTOutput&) src/./streams.h:460
          NEW_FUNC[0/1]: 0x55c934800100 in prevector<28u, unsigned char, unsigned int, int>::size() const src/./prevector.h:277
          NEW_FUNC[0/4]: 0x55c934849840 in void CScript::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./script/script.h:418
          NEW_FUNC[1/4]: 0x55c934849b40 in void Unserialize_impl<CDataStream, 28u, unsigned char>(CDataStream&, prevector<28u, unsigned char, unsigned int, int>&, unsigned char const&) src/./serialize.h:666
          NEW_FUNC[2/4]: 0x55c934849f70 in prevector<28u, unsigned char, unsigned int, int>::resize(unsigned int) src/./prevector.h:311
          NEW_FUNC[3/4]: 0x55c93485dc60 in CDataStream& CDataStream::operator>><CScript&>(CScript&) src/./streams.h:460
          NEW_FUNC[0/3]: 0x55c934800310 in prevector<28u, unsigned char, unsigned int, int>::capacity() const src/./prevector.h:295
          NEW_FUNC[1/3]: 0x55c934800d70 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) src/./prevector.h:196
          NEW_FUNC[2/3]: 0x55c934849d40 in prevector<28u, unsigned char, unsigned int, int>::resize_uninitialized(unsigned int) src/./prevector.h:381
          NEW_FUNC[0/1]: 0x55c93485ddd0 in void DeserializeHDKeypaths<CDataStream>(CDataStream&, std::vector<unsigned char, std::allocator<unsigned char> > const&, std::map<CPubKey, KeyOriginInfo, std::less<CPubKey>, std::allocator<std::pair<CPubKey const, KeyOriginInfo> > >&) src/./script/sign.h:103
  stat::number_of_executed_units: 19130
  stat::average_exec_per_sec:     1739
  stat::new_units_added:          195
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              411
  Number of unique code paths taken during fuzzing round: 64

  Tested fuzz harnesses seem to work as expected.
  ```

Top commit has no ACKs.

Tree-SHA512: baf1630a6e438d02d33c77b9e602c99546b9e8d83705e67c2749e0600039c37707cdf419cee19282f069e8d787c536ed4960f9c47e93bd0f0251495b83780ada
@maflcko maflcko merged commit a1308b7 into bitcoin:master Dec 11, 2019
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Dec 12, 2019
…ng scripts, numbers, JSON and HD keypaths (bip32)

a1308b7 tests: Add fuzzing harnesses for various JSON/univalue parsing functions (practicalswift)
e3d2bcf tests: Add fuzzing harnesses for various number parsing functions (practicalswift)
fb8c120 tests: Add ParseScript(...) (core_io) fuzzing harness (practicalswift)
074cb64 tests: Add ParseHDKeypath(...) (bip32) fuzzing harness (practicalswift)
0dc5907 tests: Add corpora suppression (FUZZERS_MISSING_CORPORA) for fuzzers missing in https://github.com/bitcoin-core/qa-assets/tree/master/fuzz_seed_corpus (practicalswift)

Pull request description:

  Add fuzzing harnesses for `DecodeRawPSBT(...)`, `ParseHDKeypath(...)`, `ParseScript(...)`, various number parsing functions and various JSON/univalue parsing functions.

  **Testing this PR**
  As usual the best way to test proposed fuzzing harnesses is to use `test_fuzzing_harnesses.sh` (bitcoin#17000) to quickly verify that the relevant code regions are triggered, that the fuzzing throughput seems reasonable, etc.

  `test_fuzzing_harnesses.sh 'psbt|hd_keypath|numbers|parse_script|univalue' 10` runs all fuzzers matching the regexp and gives them ten seconds of runtime each.

  ```
  $ CC=clang CXX=clang++ ./configure --enable-fuzz --with-sanitizers=address,fuzzer,undefined
  $ make
  $ contrib/devtools/test_fuzzing_harnesses.sh 'psbt|hd_keypath|numbers|parse_script|univalue' 10
  Testing fuzzer parse_hd_keypath during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[0/2]: 0x55bc23a76940 in ParsePrechecks(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/util/strencodings.cpp:267
          NEW_FUNC[1/2]: 0x55bc23a77300 in ParseUInt32(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int*) src/util/strencodings.cpp:309
  stat::number_of_executed_units: 34237
  stat::average_exec_per_sec:     3112
  stat::new_units_added:          113
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              282
  Number of unique code paths taken during fuzzing round: 30

  Testing fuzzer parse_numbers during 10 second(s)
  A subset of reached functions:
  stat::number_of_executed_units: 31309
  stat::average_exec_per_sec:     2846
  stat::new_units_added:          688
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              234
  Number of unique code paths taken during fuzzing round: 149

  Testing fuzzer parse_script during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[1/11]: 0x5636ff61ba00 in IsDigit(char) src/./util/strencodings.h:70
          NEW_FUNC[0/14]: 0x5636fe6c6280 in CScript::operator<<(opcodetype) src/./script/script.h:448
          NEW_FUNC[1/14]: 0x5636fe6e0290 in prevector<28u, unsigned char, unsigned int, int>::insert(prevector<28u, unsigned char, unsigned int, int>::iterator, unsigned char const&) src/./prevector.h:342
          NEW_FUNC[2/14]: 0x5636fe6e1040 in prevector<28u, unsigned char, unsigned int, int>::size() const src/./prevector.h:277
          NEW_FUNC[3/14]: 0x5636fe6e1250 in prevector<28u, unsigned char, unsigned int, int>::capacity() const src/./prevector.h:295
          NEW_FUNC[4/14]: 0x5636fe6e1cb0 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) src/./prevector.h:196
          NEW_FUNC[0/10]: 0x5636fe6c5650 in CScript::operator<<(std::vector<unsigned char, std::allocator<unsigned char> > const&) src/./script/script.h:462
          NEW_FUNC[2/10]: 0x5636fe6e0a20 in void prevector<28u, unsigned char, unsigned int, int>::insert<__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > > >(prevector<28u, unsigned char, unsigned int, int>::iterator, __gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<[32/1902]
  char> > >, __gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > >) src/./prevector.h:368
          NEW_FUNC[5/10]: 0x5636fe6e2350 in void prevector<28u, unsigned char, unsigned int, int>::fill<__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > > >(unsigned char*, __gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > >, __gnu_cxx::__normal_iterator<unsign
  ed char const*, std::vector<unsigned char, std::allocator<unsigned char> > >) src/./prevector.h:204
          NEW_FUNC[0/1]: 0x5636ff8e48b0 in IsHex(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/util/strencodings.cpp:61
          NEW_FUNC[0/2]: 0x5636fe6e1410 in prevector<28u, unsigned char, unsigned int, int>::change_capacity(unsigned int) src/./prevector.h:165
          NEW_FUNC[1/2]: 0x5636fe6e1f00 in prevector<28u, unsigned char, unsigned int, int>::indirect_ptr(int) src/./prevector.h:161
          NEW_FUNC[0/1]: 0x5636fe6e0580 in void prevector<28u, unsigned char, unsigned int, int>::insert<unsigned char*>(prevector<28u, unsigned char, unsigned int, int>::iterator, unsigned char*, unsigned char*) src/./prevector.h:368
          NEW_FUNC[0/3]: 0x5636fe85f0d0 in CScript::push_int64(long) src/./script/script.h:394
          NEW_FUNC[1/3]: 0x5636fe85f520 in prevector<28u, unsigned char, unsigned int, int>::push_back(unsigned char const&) src/./prevector.h:422
          NEW_FUNC[2/3]: 0x5636ff8ed730 in atoi64(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/util/strencodings.cpp:417
  stat::number_of_executed_units: 8153
  stat::average_exec_per_sec:     741
  stat::new_units_added:          296
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              237
  Number of unique code paths taken during fuzzing round: 98

  Testing fuzzer parse_univalue during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[0/19]: 0x560db8655950 in tinyformat::detail::formatImpl(std::ostream&, char const*, tinyformat::detail::FormatArg const*, int) src/./tinyformat.h:791
          NEW_FUNC[4/19]: 0x560db86582b0 in tinyformat::detail::printFormatStringLiteral(std::ostream&, char const*) src/./tinyformat.h:564
          NEW_FUNC[5/19]: 0x560db8658690 in tinyformat::detail::streamStateFromFormat(std::ostream&, bool&, int&, char const*, tinyformat::detail::FormatArg const*, int&, int) src/./tinyformat.h:601
          NEW_FUNC[6/19]: 0x560db865f090 in tinyformat::detail::FormatArg::format(std::ostream&, char const*, char const*, int) const src/./tinyformat.h:513
          NEW_FUNC[12/19]: 0x560db8661ba0 in void tinyformat::detail::FormatArg::formatImpl<int>(std::ostream&, char const*, char const*, int, void const*) src/./tinyformat.h:530
          NEW_FUNC[13/19]: 0x560db8661d90 in void tinyformat::formatValue<int>(std::ostream&, char const*, char const*, int, int const&) src/./tinyformat.h:317
          NEW_FUNC[14/19]: 0x560db875c8b0 in void tinyformat::detail::FormatArg::formatImpl<unsigned int>(std::ostream&, char const*, char const*, int, void const*) src/./tinyformat.h:530
          NEW_FUNC[15/19]: 0x560db875caa0 in void tinyformat::formatValue<unsigned int>(std::ostream&, char const*, char const*, int, unsigned int const&) src/./tinyformat.h:317
          NEW_FUNC[16/19]: 0x560db9473ef0 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > tinyformat::format<int, unsigned int>(char const*, int const&, unsigned int const&) src/./tinyformat.h:976
          NEW_FUNC[17/19]: 0x560db94749a0 in void tinyformat::format<int, unsigned int>(std::ostream&, char const*, int const&, unsigned int const&) src/./tinyformat.h:968
          NEW_FUNC[18/19]: 0x560db9474cf0 in tinyformat::detail::FormatListN<2>::FormatListN<int, unsigned int>(int const&, unsigned int const&) src/./tinyformat.h:885
  stat::number_of_executed_units: 14089
  stat::average_exec_per_sec:     1280
  stat::new_units_added:          135
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              356
  Number of unique code paths taken during fuzzing round: 62

  Testing fuzzer psbt_input_deserialize during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[0/46]: 0x557847ce3530 in prevector<28u, unsigned char, unsigned int, int>::~prevector() src/./prevector.h:456
          NEW_FUNC[3/46]: 0x557847cfdcf0 in prevector<28u, unsigned char, unsigned int, int>::size() const src/./prevector.h:277
          NEW_FUNC[4/46]: 0x557847cfe0c0 in prevector<28u, unsigned char, unsigned int, int>::change_capacity(unsigned int) src/./prevector.h:165
          NEW_FUNC[13/46]: 0x557847d3c890 in unsigned long ReadCompactSize<CDataStream>(CDataStream&) src/./serialize.h:290
          NEW_FUNC[14/46]: 0x557847d47b60 in prevector<28u, unsigned char, unsigned int, int>::resize(unsigned int) src/./prevector.h:311
          NEW_FUNC[16/46]: 0x557847d48800 in CTxOut::CTxOut() src/./primitives/transaction.h:140
          NEW_FUNC[17/46]: 0x557847d4b050 in CTxOut::SetNull() src/./primitives/transaction.h:155
          NEW_FUNC[18/46]: 0x557847d4b140 in CScript::clear() src/./script/script.h:563
          NEW_FUNC[19/46]: 0x557847d4ead0 in void Unserialize_impl<CDataStream, unsigned char, std::allocator<unsigned char> >(CDataStream&, std::vector<unsigned char, std::allocator<unsigned char> >&, unsigned char const&) src/./serialize.h:746
          NEW_FUNC[0/58]: 0x557847cfdf00 in prevector<28u, unsigned char, unsigned int, int>::capacity() const src/./prevector.h:295
          NEW_FUNC[1/58]: 0x557847cfe960 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) src/./prevector.h:196
          NEW_FUNC[2/58]: 0x557847cfebb0 in prevector<28u, unsigned char, unsigned int, int>::indirect_ptr(int) src/./prevector.h:161
          NEW_FUNC[3/58]: 0x557847d03990 in uint256::uint256() src/./uint256.h:123
          NEW_FUNC[0/3]: 0x557847d47430 in void CScript::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./script/script.h:418
          NEW_FUNC[1/3]: 0x557847d47730 in void Unserialize_impl<CDataStream, 28u, unsigned char>(CDataStream&, prevector<28u, unsigned char, unsigned int, int>&, unsigned char const&) src/./serialize.h:666
          NEW_FUNC[2/3]: 0x557847d60dd0 in CDataStream& CDataStream::operator>><CScript&>(CScript&) src/./streams.h:460
          NEW_FUNC[1/78]: 0x557847cffae0 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) const src/./prevector.h:197
          NEW_FUNC[2/78]: 0x557847cffd30 in prevector<28u, unsigned char, unsigned int, int>::indirect_ptr(int) const src/./prevector.h:162
          NEW_FUNC[0/1]: 0x557847d65f90 in OverrideStream<CDataStream>& OverrideStream<CDataStream>::operator>><unsigned char&>(unsigned char&) src/./streams.h:46
          NEW_FUNC[0/3]: 0x557847d470e0 in void SerReadWriteMany<CDataStream, CScript&>(CDataStream&, CSerActionUnserialize, CScript&) src/./serialize.h:989
          NEW_FUNC[1/3]: 0x557847d4ac50 in void CTxOut::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./primitives/transaction.h:149
          NEW_FUNC[2/3]: 0x557847d5f860 in void UnserializeFromVector<CDataStream, CTxOut>(CDataStream&, CTxOut&) src/./script/sign.h:90
          NEW_FUNC[0/1]: 0x557847d60840 in void UnserializeFromVector<CDataStream, int>(CDataStream&, int&) src/./script/sign.h:90
          NEW_FUNC[0/1]: 0x557847d41010 in CMutableTransaction::HasWitness() const src/./primitives/transaction.h:398
  stat::number_of_executed_units: 13615
  stat::average_exec_per_sec:     1237
  stat::new_units_added:          357
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              446
  Number of unique code paths taken during fuzzing round: 152

  Testing fuzzer psbt_output_deserialize during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[0/27]: 0x55c9347e5940 in prevector<28u, unsigned char, unsigned int, int>::~prevector() src/./prevector.h:456
          NEW_FUNC[5/27]: 0x55c93483eca0 in unsigned long ReadCompactSize<CDataStream>(CDataStream&) src/./serialize.h:290
          NEW_FUNC[6/27]: 0x55c934850ee0 in void Unserialize_impl<CDataStream, unsigned char, std::allocator<unsigned char> >(CDataStream&, std::vector<unsigned char, std::allocator<unsigned char> >&, unsigned char const&) src/./serialize.h:746
          NEW_FUNC[14/27]: 0x55c934858500 in PSBTOutput::PSBTOutput() src/./psbt.h:281
          NEW_FUNC[15/27]: 0x55c934858870 in CDataStream& CDataStream::operator>><PSBTOutput&>(PSBTOutput&) src/./streams.h:460
          NEW_FUNC[0/1]: 0x55c934800100 in prevector<28u, unsigned char, unsigned int, int>::size() const src/./prevector.h:277
          NEW_FUNC[0/4]: 0x55c934849840 in void CScript::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./script/script.h:418
          NEW_FUNC[1/4]: 0x55c934849b40 in void Unserialize_impl<CDataStream, 28u, unsigned char>(CDataStream&, prevector<28u, unsigned char, unsigned int, int>&, unsigned char const&) src/./serialize.h:666
          NEW_FUNC[2/4]: 0x55c934849f70 in prevector<28u, unsigned char, unsigned int, int>::resize(unsigned int) src/./prevector.h:311
          NEW_FUNC[3/4]: 0x55c93485dc60 in CDataStream& CDataStream::operator>><CScript&>(CScript&) src/./streams.h:460
          NEW_FUNC[0/3]: 0x55c934800310 in prevector<28u, unsigned char, unsigned int, int>::capacity() const src/./prevector.h:295
          NEW_FUNC[1/3]: 0x55c934800d70 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) src/./prevector.h:196
          NEW_FUNC[2/3]: 0x55c934849d40 in prevector<28u, unsigned char, unsigned int, int>::resize_uninitialized(unsigned int) src/./prevector.h:381
          NEW_FUNC[0/1]: 0x55c93485ddd0 in void DeserializeHDKeypaths<CDataStream>(CDataStream&, std::vector<unsigned char, std::allocator<unsigned char> > const&, std::map<CPubKey, KeyOriginInfo, std::less<CPubKey>, std::allocator<std::pair<CPubKey const, KeyOriginInfo> > >&) src/./script/sign.h:103
  stat::number_of_executed_units: 19130
  stat::average_exec_per_sec:     1739
  stat::new_units_added:          195
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              411
  Number of unique code paths taken during fuzzing round: 64

  Tested fuzz harnesses seem to work as expected.
  ```

Top commit has no ACKs.

Tree-SHA512: baf1630a6e438d02d33c77b9e602c99546b9e8d83705e67c2749e0600039c37707cdf419cee19282f069e8d787c536ed4960f9c47e93bd0f0251495b83780ada
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Oct 28, 2020
…JSON and HD keypaths (bip32)

Summary:
```
Add fuzzing harnesses for DecodeRawPSBT(...), ParseHDKeypath(...),
ParseScript(...), various number parsing functions and various
JSON/univalue parsing functions.
```

Backport of core [[bitcoin/bitcoin#17050 | PR17050]].

The missing corpora change is not relevant for us.

Test Plan:
  ninja bitcoin-fuzzers
  ./test/fuzz/test_runner.py <path_to_corpus>

Reviewers: #bitcoin_abc, deadalnix

Reviewed By: #bitcoin_abc, deadalnix

Subscribers: deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D8149
sidhujag pushed a commit to syscoin-core/syscoin that referenced this pull request Nov 10, 2020
…ng scripts, numbers, JSON and HD keypaths (bip32)

a1308b7 tests: Add fuzzing harnesses for various JSON/univalue parsing functions (practicalswift)
e3d2bcf tests: Add fuzzing harnesses for various number parsing functions (practicalswift)
fb8c120 tests: Add ParseScript(...) (core_io) fuzzing harness (practicalswift)
074cb64 tests: Add ParseHDKeypath(...) (bip32) fuzzing harness (practicalswift)
0dc5907 tests: Add corpora suppression (FUZZERS_MISSING_CORPORA) for fuzzers missing in https://github.com/bitcoin-core/qa-assets/tree/master/fuzz_seed_corpus (practicalswift)

Pull request description:

  Add fuzzing harnesses for `DecodeRawPSBT(...)`, `ParseHDKeypath(...)`, `ParseScript(...)`, various number parsing functions and various JSON/univalue parsing functions.

  **Testing this PR**
  As usual the best way to test proposed fuzzing harnesses is to use `test_fuzzing_harnesses.sh` (bitcoin#17000) to quickly verify that the relevant code regions are triggered, that the fuzzing throughput seems reasonable, etc.

  `test_fuzzing_harnesses.sh 'psbt|hd_keypath|numbers|parse_script|univalue' 10` runs all fuzzers matching the regexp and gives them ten seconds of runtime each.

  ```
  $ CC=clang CXX=clang++ ./configure --enable-fuzz --with-sanitizers=address,fuzzer,undefined
  $ make
  $ contrib/devtools/test_fuzzing_harnesses.sh 'psbt|hd_keypath|numbers|parse_script|univalue' 10
  Testing fuzzer parse_hd_keypath during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[0/2]: 0x55bc23a76940 in ParsePrechecks(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/util/strencodings.cpp:267
          NEW_FUNC[1/2]: 0x55bc23a77300 in ParseUInt32(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int*) src/util/strencodings.cpp:309
  stat::number_of_executed_units: 34237
  stat::average_exec_per_sec:     3112
  stat::new_units_added:          113
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              282
  Number of unique code paths taken during fuzzing round: 30

  Testing fuzzer parse_numbers during 10 second(s)
  A subset of reached functions:
  stat::number_of_executed_units: 31309
  stat::average_exec_per_sec:     2846
  stat::new_units_added:          688
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              234
  Number of unique code paths taken during fuzzing round: 149

  Testing fuzzer parse_script during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[1/11]: 0x5636ff61ba00 in IsDigit(char) src/./util/strencodings.h:70
          NEW_FUNC[0/14]: 0x5636fe6c6280 in CScript::operator<<(opcodetype) src/./script/script.h:448
          NEW_FUNC[1/14]: 0x5636fe6e0290 in prevector<28u, unsigned char, unsigned int, int>::insert(prevector<28u, unsigned char, unsigned int, int>::iterator, unsigned char const&) src/./prevector.h:342
          NEW_FUNC[2/14]: 0x5636fe6e1040 in prevector<28u, unsigned char, unsigned int, int>::size() const src/./prevector.h:277
          NEW_FUNC[3/14]: 0x5636fe6e1250 in prevector<28u, unsigned char, unsigned int, int>::capacity() const src/./prevector.h:295
          NEW_FUNC[4/14]: 0x5636fe6e1cb0 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) src/./prevector.h:196
          NEW_FUNC[0/10]: 0x5636fe6c5650 in CScript::operator<<(std::vector<unsigned char, std::allocator<unsigned char> > const&) src/./script/script.h:462
          NEW_FUNC[2/10]: 0x5636fe6e0a20 in void prevector<28u, unsigned char, unsigned int, int>::insert<__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > > >(prevector<28u, unsigned char, unsigned int, int>::iterator, __gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<[32/1902]
  char> > >, __gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > >) src/./prevector.h:368
          NEW_FUNC[5/10]: 0x5636fe6e2350 in void prevector<28u, unsigned char, unsigned int, int>::fill<__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > > >(unsigned char*, __gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > >, __gnu_cxx::__normal_iterator<unsign
  ed char const*, std::vector<unsigned char, std::allocator<unsigned char> > >) src/./prevector.h:204
          NEW_FUNC[0/1]: 0x5636ff8e48b0 in IsHex(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/util/strencodings.cpp:61
          NEW_FUNC[0/2]: 0x5636fe6e1410 in prevector<28u, unsigned char, unsigned int, int>::change_capacity(unsigned int) src/./prevector.h:165
          NEW_FUNC[1/2]: 0x5636fe6e1f00 in prevector<28u, unsigned char, unsigned int, int>::indirect_ptr(int) src/./prevector.h:161
          NEW_FUNC[0/1]: 0x5636fe6e0580 in void prevector<28u, unsigned char, unsigned int, int>::insert<unsigned char*>(prevector<28u, unsigned char, unsigned int, int>::iterator, unsigned char*, unsigned char*) src/./prevector.h:368
          NEW_FUNC[0/3]: 0x5636fe85f0d0 in CScript::push_int64(long) src/./script/script.h:394
          NEW_FUNC[1/3]: 0x5636fe85f520 in prevector<28u, unsigned char, unsigned int, int>::push_back(unsigned char const&) src/./prevector.h:422
          NEW_FUNC[2/3]: 0x5636ff8ed730 in atoi64(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/util/strencodings.cpp:417
  stat::number_of_executed_units: 8153
  stat::average_exec_per_sec:     741
  stat::new_units_added:          296
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              237
  Number of unique code paths taken during fuzzing round: 98

  Testing fuzzer parse_univalue during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[0/19]: 0x560db8655950 in tinyformat::detail::formatImpl(std::ostream&, char const*, tinyformat::detail::FormatArg const*, int) src/./tinyformat.h:791
          NEW_FUNC[4/19]: 0x560db86582b0 in tinyformat::detail::printFormatStringLiteral(std::ostream&, char const*) src/./tinyformat.h:564
          NEW_FUNC[5/19]: 0x560db8658690 in tinyformat::detail::streamStateFromFormat(std::ostream&, bool&, int&, char const*, tinyformat::detail::FormatArg const*, int&, int) src/./tinyformat.h:601
          NEW_FUNC[6/19]: 0x560db865f090 in tinyformat::detail::FormatArg::format(std::ostream&, char const*, char const*, int) const src/./tinyformat.h:513
          NEW_FUNC[12/19]: 0x560db8661ba0 in void tinyformat::detail::FormatArg::formatImpl<int>(std::ostream&, char const*, char const*, int, void const*) src/./tinyformat.h:530
          NEW_FUNC[13/19]: 0x560db8661d90 in void tinyformat::formatValue<int>(std::ostream&, char const*, char const*, int, int const&) src/./tinyformat.h:317
          NEW_FUNC[14/19]: 0x560db875c8b0 in void tinyformat::detail::FormatArg::formatImpl<unsigned int>(std::ostream&, char const*, char const*, int, void const*) src/./tinyformat.h:530
          NEW_FUNC[15/19]: 0x560db875caa0 in void tinyformat::formatValue<unsigned int>(std::ostream&, char const*, char const*, int, unsigned int const&) src/./tinyformat.h:317
          NEW_FUNC[16/19]: 0x560db9473ef0 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > tinyformat::format<int, unsigned int>(char const*, int const&, unsigned int const&) src/./tinyformat.h:976
          NEW_FUNC[17/19]: 0x560db94749a0 in void tinyformat::format<int, unsigned int>(std::ostream&, char const*, int const&, unsigned int const&) src/./tinyformat.h:968
          NEW_FUNC[18/19]: 0x560db9474cf0 in tinyformat::detail::FormatListN<2>::FormatListN<int, unsigned int>(int const&, unsigned int const&) src/./tinyformat.h:885
  stat::number_of_executed_units: 14089
  stat::average_exec_per_sec:     1280
  stat::new_units_added:          135
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              356
  Number of unique code paths taken during fuzzing round: 62

  Testing fuzzer psbt_input_deserialize during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[0/46]: 0x557847ce3530 in prevector<28u, unsigned char, unsigned int, int>::~prevector() src/./prevector.h:456
          NEW_FUNC[3/46]: 0x557847cfdcf0 in prevector<28u, unsigned char, unsigned int, int>::size() const src/./prevector.h:277
          NEW_FUNC[4/46]: 0x557847cfe0c0 in prevector<28u, unsigned char, unsigned int, int>::change_capacity(unsigned int) src/./prevector.h:165
          NEW_FUNC[13/46]: 0x557847d3c890 in unsigned long ReadCompactSize<CDataStream>(CDataStream&) src/./serialize.h:290
          NEW_FUNC[14/46]: 0x557847d47b60 in prevector<28u, unsigned char, unsigned int, int>::resize(unsigned int) src/./prevector.h:311
          NEW_FUNC[16/46]: 0x557847d48800 in CTxOut::CTxOut() src/./primitives/transaction.h:140
          NEW_FUNC[17/46]: 0x557847d4b050 in CTxOut::SetNull() src/./primitives/transaction.h:155
          NEW_FUNC[18/46]: 0x557847d4b140 in CScript::clear() src/./script/script.h:563
          NEW_FUNC[19/46]: 0x557847d4ead0 in void Unserialize_impl<CDataStream, unsigned char, std::allocator<unsigned char> >(CDataStream&, std::vector<unsigned char, std::allocator<unsigned char> >&, unsigned char const&) src/./serialize.h:746
          NEW_FUNC[0/58]: 0x557847cfdf00 in prevector<28u, unsigned char, unsigned int, int>::capacity() const src/./prevector.h:295
          NEW_FUNC[1/58]: 0x557847cfe960 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) src/./prevector.h:196
          NEW_FUNC[2/58]: 0x557847cfebb0 in prevector<28u, unsigned char, unsigned int, int>::indirect_ptr(int) src/./prevector.h:161
          NEW_FUNC[3/58]: 0x557847d03990 in uint256::uint256() src/./uint256.h:123
          NEW_FUNC[0/3]: 0x557847d47430 in void CScript::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./script/script.h:418
          NEW_FUNC[1/3]: 0x557847d47730 in void Unserialize_impl<CDataStream, 28u, unsigned char>(CDataStream&, prevector<28u, unsigned char, unsigned int, int>&, unsigned char const&) src/./serialize.h:666
          NEW_FUNC[2/3]: 0x557847d60dd0 in CDataStream& CDataStream::operator>><CScript&>(CScript&) src/./streams.h:460
          NEW_FUNC[1/78]: 0x557847cffae0 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) const src/./prevector.h:197
          NEW_FUNC[2/78]: 0x557847cffd30 in prevector<28u, unsigned char, unsigned int, int>::indirect_ptr(int) const src/./prevector.h:162
          NEW_FUNC[0/1]: 0x557847d65f90 in OverrideStream<CDataStream>& OverrideStream<CDataStream>::operator>><unsigned char&>(unsigned char&) src/./streams.h:46
          NEW_FUNC[0/3]: 0x557847d470e0 in void SerReadWriteMany<CDataStream, CScript&>(CDataStream&, CSerActionUnserialize, CScript&) src/./serialize.h:989
          NEW_FUNC[1/3]: 0x557847d4ac50 in void CTxOut::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./primitives/transaction.h:149
          NEW_FUNC[2/3]: 0x557847d5f860 in void UnserializeFromVector<CDataStream, CTxOut>(CDataStream&, CTxOut&) src/./script/sign.h:90
          NEW_FUNC[0/1]: 0x557847d60840 in void UnserializeFromVector<CDataStream, int>(CDataStream&, int&) src/./script/sign.h:90
          NEW_FUNC[0/1]: 0x557847d41010 in CMutableTransaction::HasWitness() const src/./primitives/transaction.h:398
  stat::number_of_executed_units: 13615
  stat::average_exec_per_sec:     1237
  stat::new_units_added:          357
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              446
  Number of unique code paths taken during fuzzing round: 152

  Testing fuzzer psbt_output_deserialize during 10 second(s)
  A subset of reached functions:
          NEW_FUNC[0/27]: 0x55c9347e5940 in prevector<28u, unsigned char, unsigned int, int>::~prevector() src/./prevector.h:456
          NEW_FUNC[5/27]: 0x55c93483eca0 in unsigned long ReadCompactSize<CDataStream>(CDataStream&) src/./serialize.h:290
          NEW_FUNC[6/27]: 0x55c934850ee0 in void Unserialize_impl<CDataStream, unsigned char, std::allocator<unsigned char> >(CDataStream&, std::vector<unsigned char, std::allocator<unsigned char> >&, unsigned char const&) src/./serialize.h:746
          NEW_FUNC[14/27]: 0x55c934858500 in PSBTOutput::PSBTOutput() src/./psbt.h:281
          NEW_FUNC[15/27]: 0x55c934858870 in CDataStream& CDataStream::operator>><PSBTOutput&>(PSBTOutput&) src/./streams.h:460
          NEW_FUNC[0/1]: 0x55c934800100 in prevector<28u, unsigned char, unsigned int, int>::size() const src/./prevector.h:277
          NEW_FUNC[0/4]: 0x55c934849840 in void CScript::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./script/script.h:418
          NEW_FUNC[1/4]: 0x55c934849b40 in void Unserialize_impl<CDataStream, 28u, unsigned char>(CDataStream&, prevector<28u, unsigned char, unsigned int, int>&, unsigned char const&) src/./serialize.h:666
          NEW_FUNC[2/4]: 0x55c934849f70 in prevector<28u, unsigned char, unsigned int, int>::resize(unsigned int) src/./prevector.h:311
          NEW_FUNC[3/4]: 0x55c93485dc60 in CDataStream& CDataStream::operator>><CScript&>(CScript&) src/./streams.h:460
          NEW_FUNC[0/3]: 0x55c934800310 in prevector<28u, unsigned char, unsigned int, int>::capacity() const src/./prevector.h:295
          NEW_FUNC[1/3]: 0x55c934800d70 in prevector<28u, unsigned char, unsigned int, int>::item_ptr(int) src/./prevector.h:196
          NEW_FUNC[2/3]: 0x55c934849d40 in prevector<28u, unsigned char, unsigned int, int>::resize_uninitialized(unsigned int) src/./prevector.h:381
          NEW_FUNC[0/1]: 0x55c93485ddd0 in void DeserializeHDKeypaths<CDataStream>(CDataStream&, std::vector<unsigned char, std::allocator<unsigned char> > const&, std::map<CPubKey, KeyOriginInfo, std::less<CPubKey>, std::allocator<std::pair<CPubKey const, KeyOriginInfo> > >&) src/./script/sign.h:103
  stat::number_of_executed_units: 19130
  stat::average_exec_per_sec:     1739
  stat::new_units_added:          195
  stat::slowest_unit_time_sec:    0
  stat::peak_rss_mb:              411
  Number of unique code paths taken during fuzzing round: 64

  Tested fuzz harnesses seem to work as expected.
  ```

Top commit has no ACKs.

Tree-SHA512: baf1630a6e438d02d33c77b9e602c99546b9e8d83705e67c2749e0600039c37707cdf419cee19282f069e8d787c536ed4960f9c47e93bd0f0251495b83780ada
@practicalswift practicalswift deleted the fuzzers branch April 10, 2021 19:39
kwvg added a commit to kwvg/dash that referenced this pull request Feb 27, 2022
…pts, numbers, JSON and HD keypaths (bip32)
kwvg added a commit to kwvg/dash that referenced this pull request Feb 27, 2022
…pts, numbers, JSON and HD keypaths (bip32)
kwvg added a commit to kwvg/dash that referenced this pull request Feb 28, 2022
…pts, numbers, JSON and HD keypaths (bip32)
kwvg added a commit to kwvg/dash that referenced this pull request Feb 28, 2022
…pts, numbers, JSON and HD keypaths (bip32)
kwvg added a commit to kwvg/dash that referenced this pull request Feb 28, 2022
…pts, numbers, JSON and HD keypaths (bip32)
kwvg added a commit to kwvg/dash that referenced this pull request Mar 13, 2022
…pts, numbers, JSON and HD keypaths (bip32)
kwvg added a commit to kwvg/dash that referenced this pull request Mar 24, 2022
…pts, numbers, JSON and HD keypaths (bip32)
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants