fuzz: Avoid OOM in transaction fuzz target #27921
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. |
|
This may also help with: |
Also fix bug where the json object is reused between two calls.
|
Checked locally that this fixes the CI task |
dergoegge
approved these changes
Jun 21, 2023
|
reconfirmed that this OOM's, and ran into another issue while doing so: Run transaction with args ['/home/ubuntu/ci_scratch/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/fuzz/fuzz', '-runs=1', PosixPath('/home/ubuntu/ci_scratch/ci/scratch/qa-assets/fuzz_seed_corpus/transaction')]INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2714625807
INFO: Loaded 1 modules (406370 inline 8-bit counters): 406370 [0x55a28ea008e0, 0x55a28ea63c42),
INFO: Loaded 1 PC tables (406370 PCs): 406370 [0x55a28ea63c48,0x55a28f097268),
INFO: 3510 files found in /home/ubuntu/ci_scratch/ci/scratch/qa-assets/fuzz_seed_corpus/transaction
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
INFO: seed corpus: files: 3510 min: 1b max: 1855780b total: 192527534b rss: 112Mb
#1024 pulse cov: 3002 ft: 6917 corp: 312/16Kb exec/s: 341 rss: 136Mb
#2048 pulse cov: 3427 ft: 14596 corp: 845/367Kb exec/s: 75 rss: 138Mb
Slowest unit: 11 s:
artifact_prefix='./'; Test unit written to ./slow-unit-be16ac19bb5b0840ca16f3dfd3280eb9a3031e73
Slowest unit: 15 s:
artifact_prefix='./'; Test unit written to ./slow-unit-96435df971eba2c9c4bb075d3f3528eb3d82cd46
Slowest unit: 23 s:
artifact_prefix='./'; Test unit written to ./slow-unit-639c027d00c0206287054e862ca5f6c9fade1320
Slowest unit: 35 s:
artifact_prefix='./'; Test unit written to ./slow-unit-9ee82a6a83f031c81d2512050ee5ecdc9f3eb527
Slowest unit: 45 s:
artifact_prefix='./'; Test unit written to ./slow-unit-91ee2766360efce6d700daa1eacd872616f6e434
Slowest unit: 72 s:
artifact_prefix='./'; Test unit written to ./slow-unit-ae198ef8388da086d18ebbbcb42cca642faceda7
==30091== ERROR: libFuzzer: out-of-memory (used: 2075Mb; limit: 2048Mb)
To change the out-of-memory limit use -rss_limit_mb=<N>
MS: 0 ; base unit: 0000000000000000000000000000000000000000
artifact_prefix='./'; Test unit written to ./oom-3ab0860b24a11833b3dec26b47a0a9b8fcc5fcd4
SUMMARY: libFuzzer: out-of-memory
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2714625807
INFO: Loaded 1 modules (406370 inline 8-bit counters): 406370 [0x55a28ea008e0, 0x55a28ea63c42),
INFO: Loaded 1 PC tables (406370 PCs): 406370 [0x55a28ea63c48,0x55a28f097268),
INFO: 3510 files found in /home/ubuntu/ci_scratch/ci/scratch/qa-assets/fuzz_seed_corpus/transaction
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
INFO: seed corpus: files: 3510 min: 1b max: 1855780b total: 192527534b rss: 112Mb
#1024 pulse cov: 3002 ft: 6917 corp: 312/16Kb exec/s: 341 rss: 136Mb
#2048 pulse cov: 3427 ft: 14596 corp: 845/367Kb exec/s: 75 rss: 138Mb
Slowest unit: 11 s:
artifact_prefix='./'; Test unit written to ./slow-unit-be16ac19bb5b0840ca16f3dfd3280eb9a3031e73
Slowest unit: 15 s:
artifact_prefix='./'; Test unit written to ./slow-unit-96435df971eba2c9c4bb075d3f3528eb3d82cd46
Slowest unit: 23 s:
artifact_prefix='./'; Test unit written to ./slow-unit-639c027d00c0206287054e862ca5f6c9fade1320
Slowest unit: 35 s:
artifact_prefix='./'; Test unit written to ./slow-unit-9ee82a6a83f031c81d2512050ee5ecdc9f3eb527
Slowest unit: 45 s:
artifact_prefix='./'; Test unit written to ./slow-unit-91ee2766360efce6d700daa1eacd872616f6e434
Slowest unit: 72 s:
artifact_prefix='./'; Test unit written to ./slow-unit-ae198ef8388da086d18ebbbcb42cca642faceda7
==30091== ERROR: libFuzzer: out-of-memory (used: 2075Mb; limit: 2048Mb)
To change the out-of-memory limit use -rss_limit_mb=<N>
MS: 0 ; base unit: 0000000000000000000000000000000000000000
artifact_prefix='./'; Test unit written to ./oom-3ab0860b24a11833b3dec26b47a0a9b8fcc5fcd4
SUMMARY: libFuzzer: out-of-memory
Traceback (most recent call last):
File "/home/ubuntu/ci_scratch/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/fuzz/test_runner.py", line 298, in run_once
result.check_returncode()
File "/usr/lib/python3.10/subprocess.py", line 456, in check_returncode
raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['/home/ubuntu/ci_scratch/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/fuzz/fuzz', '-runs=1', PosixPath('/home/ubuntu/ci_scratch/ci/scratch/qa-assets/fuzz_seed_corpus/transaction')]' returned non-zero exit status 71.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/ci_scratch/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/fuzz/test_runner.py", line 322, in <module>
main()
File "/home/ubuntu/ci_scratch/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/fuzz/test_runner.py", line 183, in main
run_once(
File "/home/ubuntu/ci_scratch/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/fuzz/test_runner.py", line 304, in run_once
logging.info("Target \"{}\" failed with exit code {}".format(" ".join(result.args), e.returncode))
TypeError: sequence item 2: expected str instance, PosixPath found
real 195m7.994s |
" ".join(result.args)
TypeError: sequence item 2: expected str instance, PosixPath foundYeah, you can replace this with |
|
An alternative to avoid TypeError would be to rewrite the fuzz/test_runner script to a rust script from python. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To test:
FUZZ=transaction /usr/bin/time -f '%Us %MkB' ./src/test/fuzz/fuzz ../btc_qa_assets/fuzz_seed_corpus/transaction/9dc22b51df0af05ee5a595beefb0ce291feb6b99Before:
0.72s 249636kBAfter:
0.30s 92128kB