Skip to content

[tests] Speed up fuzzing by ~200x when using afl-fuzz #10415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 22, 2017
Merged

[tests] Speed up fuzzing by ~200x when using afl-fuzz #10415

merged 1 commit into from
May 22, 2017

Conversation

practicalswift
Copy link
Contributor

@practicalswift practicalswift commented May 17, 2017 •
edited
Loading

Enable the afl-clang-fast++ features deferred forkserver (__AFL_INIT) and persistent mode (__AFL_LOOP(1000)).

Before this patch:

$ afl-fuzz -i input -o output -m512 -- src/test/test_bitcoin_fuzzy
[*] Validating target binary...
[!] WARNING: The target binary is pretty slow! See /usr/local/share/doc/afl/perf_tips.txt.
[+] Here are some useful stats:

    Test case count : 1 favored, 0 variable, 1 total
       Bitmap range : 1072 to 1072 bits (average: 1072.00 bits)
        Exec timing : 20.4k to 20.4k us (average: 20.4k us)
…
exec speed : 57.58/sec (slow!)
exec speed : 48.35/sec (slow!)
exec speed : 53.78/sec (slow!)

After this patch:

$ afl-fuzz -i input -o output -m512 -- src/test/test_bitcoin_fuzzy
[*] Validating target binary...
[+] Persistent mode binary detected.
[+] Deferred forkserver binary detected.
[+] Here are some useful stats:

    Test case count : 1 favored, 0 variable, 1 total
       Bitmap range : 24 to 24 bits (average: 24.00 bits)
        Exec timing : 114 to 114 us (average: 114 us)
…
exec speed : 15.9k/sec
exec speed : 13.1k/sec
exec speed : 15.1k/sec

For more fuzzing discussions, see #10364.

@fanquake fanquake added the Tests label May 17, 2017
@practicalswift practicalswift force-pushed the fast-afl-fuzzing branch 2 times, most recently from eecdfaa to 68c4671 Compare May 17, 2017 11:01
@practicalswift practicalswift changed the title [fuzz] Speed up fuzzing by ~200x when using afl-fuzz [test] Speed up fuzzing by ~200x when using afl-fuzz May 17, 2017
@laanwj
Copy link
Member

laanwj commented May 17, 2017

Interesting. Concept ACK.

@practicalswift practicalswift changed the title [test] Speed up fuzzing by ~200x when using afl-fuzz [tests] Speed up fuzzing by ~200x when using afl-fuzz May 18, 2017
@fanquake
Copy link
Member

I'm currently seeing no speedup when using this new code. I'm wondering if it's because I have to run with AFL_NO_FORKSRV on OS X (I haven't been able to make afl-fuzz run without it being set) ?

Or does this require using afl-clang-fast and afl-clang-fast++ in place of afl-clang and afl-clang++ to see the speed improvements?

I'm seeing:

[*] Attempting dry run with 'id:000566,orig:id:000958,sync:39,src:000955'...
    len = 7368, map size = 1430, exec speed = 15345 us
[+] All test cases processed.

[!] WARNING: The target binary is pretty slow! See /usr/local/Cellar/afl-fuzz/2.41b/share/doc/afl/perf_tips.txt.
[+] Here are some useful stats:

    Test case count : 141 favored, 4 variable, 567 total
       Bitmap range : 995 to 1684 bits (average: 1240.06 bits)
        Exec timing : 13.9k to 90.9k us (average: 15.6k us)

[*] No -t option specified, so I'll use exec timeout of 100 ms.
[+] All set and ready to roll!

@practicalswift
Copy link
Contributor Author

practicalswift commented May 18, 2017 •
edited
Loading

@fanquake The binary you created is not instrumented correctly (the messages Persistent mode binary detected and Deferred forkserver binary detected are missing). You're correct that the instrumentation needed for this speedup requires using afl-clang-fast and afl-clang-fast++ in place of afl-clang and afl-clang++. See https://github.com/mirrorer/afl/tree/master/llvm_mode for details :-)

@fanquake
Copy link
Member

fanquake commented May 19, 2017 •
edited
Loading

In that case, the code changes should be accompanied by a new? section in the fuzzing documentation, outlining how to setup and run the much faster fuzzing. I'll retest this shortly.

@practicalswift
[test] Speed up fuzzing by ~200x when using afl-fuzz
693247b 
Enable the `afl-clang-fast++` features deferred forkserver (`__AFL_INIT`) and persistent mode (`__AFL_LOOP(1000)`).

Before this patch:

```
$ afl-fuzz -i input -o output -m512 -- src/test/test_bitcoin_fuzzy
[*] Validating target binary...
[!] WARNING: The target binary is pretty slow! See /usr/local/share/doc/afl/perf_tips.txt.
[+] Here are some useful stats:

    Test case count : 1 favored, 0 variable, 1 total
       Bitmap range : 1072 to 1072 bits (average: 1072.00 bits)
        Exec timing : 20.4k to 20.4k us (average: 20.4k us)
…
exec speed : 57.58/sec (slow!)
exec speed : 48.35/sec (slow!)
exec speed : 53.78/sec (slow!)
```

After this patch:

```
$ afl-fuzz -i input -o output -m512 -- src/test/test_bitcoin_fuzzy
[*] Validating target binary...
[+] Persistent mode binary detected.
[+] Deferred forkserver binary detected.
[+] Here are some useful stats:

    Test case count : 1 favored, 0 variable, 1 total
       Bitmap range : 24 to 24 bits (average: 24.00 bits)
        Exec timing : 114 to 114 us (average: 114 us)
…
exec speed : 15.9k/sec
exec speed : 13.1k/sec
exec speed : 15.1k/sec
```
@practicalswift
Copy link
Contributor Author

@fanquake Good point! Documentation and comments added. Looks good? :-)

@practicalswift practicalswift mentioned this pull request May 22, 2017
Merged
@practicalswift
Copy link
Contributor Author

Related pull request: #10440 ("Add libFuzzer support").

@laanwj
Copy link
Member

laanwj commented May 22, 2017

@fanquake thanks for testing
@practicalswift thanks for adding documentation
utACK 693247b

@laanwj laanwj merged commit 693247b into bitcoin:master May 22, 2017
laanwj added a commit that referenced this pull request May 22, 2017
@laanwj
Merge #10415: [tests] Speed up fuzzing by ~200x when using afl-fuzz
217b416 
693247b [test] Speed up fuzzing by ~200x when using afl-fuzz (practicalswift)

Tree-SHA512: 95922fc2616b8cb00dd531ed1140a52bbda4e04292dd8c1c60a8f49dbf6ccb797a18b61180b3fb68d695456b478a1f5ae7fda47e8ecee41dd65555487aef40a3
@practicalswift practicalswift mentioned this pull request Aug 16, 2017
Closed
@practicalswift practicalswift mentioned this pull request Oct 4, 2017
Merged
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Aug 24, 2019
@laanwj @PastaPastaPasta
Merge bitcoin#10415: [tests] Speed up fuzzing by ~200x when using afl…
122da98 
…-fuzz

693247b [test] Speed up fuzzing by ~200x when using afl-fuzz (practicalswift)

Tree-SHA512: 95922fc2616b8cb00dd531ed1140a52bbda4e04292dd8c1c60a8f49dbf6ccb797a18b61180b3fb68d695456b478a1f5ae7fda47e8ecee41dd65555487aef40a3
barrystyle pushed a commit to PACGlobalOfficial/PAC that referenced this pull request Jan 22, 2020
@laanwj
Merge bitcoin#10415: [tests] Speed up fuzzing by ~200x when using afl…
bd007fe 
…-fuzz

693247b [test] Speed up fuzzing by ~200x when using afl-fuzz (practicalswift)

Tree-SHA512: 95922fc2616b8cb00dd531ed1140a52bbda4e04292dd8c1c60a8f49dbf6ccb797a18b61180b3fb68d695456b478a1f5ae7fda47e8ecee41dd65555487aef40a3
@furszy furszy mentioned this pull request Mar 17, 2021
Merged
@practicalswift practicalswift deleted the fast-afl-fuzzing branch April 10, 2021 19:31
random-zebra added a commit to PIVX-Project/PIVX that referenced this pull request May 28, 2021
@random-zebra
Merge #2252: Fuzzing framework support
44b5327 
d059544 [Build] fuzz target, change LIBBITCOIN_ZEROCOIN link order. (furszy)
2396e6b [fuzz] Add ContextualCheckTransaction call to transaction target. (furszy)
f0887a0 Fuzzing documentation "PIVX-fication" (furszy)
9631f46 [doc] add sanitizers documentation in developer-notes.md (furszy)
70a0ace tests: Test serialisation as part of deserialisation fuzzing. Test round-trip equality where possible. Avoid code repetition. (practicalswift)
e1b92b6 ignore new fuzz targets gitignore (furszy)
d058d8c tests: Add deserialization fuzzing harnesses (furszy)
e1f666c tests: Remove TRANSACTION_DESERIALIZE (replaced by transaction fuzzer) (practicalswift)
b5f291c tests: Add fuzzing harness for CheckTransaction(...), IsStandardTx(...) and other CTransaction related functions (furszy)
3205871 fuzz: Remove option --export_coverage from test_runner (MarcoFalke)
52693ee fuzz: Add option to merge input dir to test runner (MarcoFalke)
2b4f8aa doc: Remove --disable-ccache from docs (MarcoFalke)
b54b1d6 tests: Improve test runner output in case of target errors (practicalswift)
cd6134f test: Log output even if fuzzer failed (MarcoFalke)
48cd0c8 doc: Improve fuzzing docs for macOS users (Fabian Jahr)
d642b67 [Build] Do not disable wallet when fuzz is enabled. (furszy)
c3447b5 Update doc and CI config (qmma)
1266d3e Disable other targets when enable-fuzz is set (qmma)
f28ac9a build: Allow to configure --with-sanitizers=fuzzer (MarcoFalke)
425742c fuzz: test_runner: Better error message when built with afl (MarcoFalke)
541f442 qa: Add test/fuzz/test_runner.py (MarcoFalke)
89fe5b2 Add missing LIBBITCOIN_ZMQ to test target (furszy)
58dbe79 add fuzzing binaries to gitignore. (furszy)
393a126 fuzz: Move deserialize tests to test/fuzz/deserialize.cpp (MarcoFalke)
a568df5 test: Build fuzz targets into separate executables (furszy)
d5dddde [test] fuzz: make test_one_input return void (MarcoFalke)
2e4ec58 [fuzzing] initialize chain params by default. (furszy)
08d8ebe [tests] Add libFuzzer support. (practicalswift)
84f72da [test] Speed up fuzzing by ~200x when using afl-fuzz (practicalswift)
faf2be6 Init ECC context for test_bitcoin_fuzzy. (Gregory Maxwell)
11150df Make fuzzer actually test CTxOutCompressor (Pieter Wuille)
d6f6a85 doc: Add bare-bones documentation for fuzzing (Wladimir J. van der Laan)
5c3b550 Simple fuzzing framework (pstratem)

Pull request description:

  As the title says, adding fuzzing framework support so we can start getting serious on this area as well.

  Adapted the following PRs:

  * bitcoin#9172.
  * bitcoin#9354.
  * bitcoin#9691.
  * bitcoin#10415.
  * bitcoin#10440.
  * bitcoin#15043.
  * bitcoin#15047.
  * bitcoin#15295.
  * bitcoin#15399 (fabcfa5 only).
  * bitcoin#16338.
  * bitcoin#17051.
  * bitcoin#17076.
  * bitcoin#17225.
  * bitcoin#17942.
  * bitcoin#16236 (only fa35c42).
  * bitcoin#18166 (only f2472f6).
  * bitcoin#18300.
  * And.. probably will go further and continue adapting more PRs..

ACKs for top commit:
  random-zebra:
    utACK d059544 and merging...

Tree-SHA512: c0b05bca47bf99bafd8abf1453c5636fe05df75f16d0e9c750368ea2aed8142f0b28d28af1d23468b8829188412a80fd3b7bdbbda294b940d78aec80c1c7d03a
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Aug 18, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
Tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@practicalswift @laanwj @fanquake