Adds fuzzing preset #7566

nathaniel-brough · 2023-05-12T23:09:47Z

Partial fix for #7552

nathaniel-brough · 2023-05-12T23:18:21Z

@steven-johnson So I was able to configure this fuzzing build correctly using;

cmake . --preset linux-x64-fuzzer

I did not need to set the CFLAGS/CXXFLAGS here and tested with those vars unset.

Hopefully this is a step in the right direction to addressing #7552. I'm going to follow up with some documentation. NOTE: that this should work on macos (even though the preset says linux). If I can confirm that it builds correctly I might just rename the preset without the "linux-" prefix. But I don't have access to a mac to test with.

Closes: halide#7552

steven-johnson

Various minor nits in the README, but the bigger issue is that the build instructions don't work for me :-/

steven-johnson · 2023-05-15T16:58:21Z

README_fuzz_testing.md

+your disk.
+
+Up until this point the fuzzer has only been running on a single core. To
+speed things up a little, let's run the fuzzer in parrelell across all


steven-johnson · 2023-05-15T16:58:46Z

README_fuzz_testing.md

+Halide has a set of fuzz-testing harnesses that can be used to find those
+tricky to find, edge cases and bugs that would otherwise not be caught
+by a regular unit-testing suite. At the moment these fuzz-tests are housed
+in the `//test/fuzz` directory. The fuzz testing suite use the common,


test/fuzz (no initial //)

steven-johnson · 2023-05-15T16:59:11Z

README_fuzz_testing.md

+[libfuzzer](https://www.llvm.org/docs/LibFuzzer.html) interface for fuzz-tests.
+
+## Building fuzz tests
+Fuzz testing requires specific instrumentation across the entire build,


grammar nit: build; rather than build,

steven-johnson · 2023-05-15T16:59:32Z

README_fuzz_testing.md

+## Using the fuzz-harnesses
+Fuzz-testing harnesses are a little different to a more traditional unit-test
+and would don't have a definitive end of test. In other words a fuzz test will
+run;


grammar nit: run: rather than run;

steven-johnson · 2023-05-15T17:00:04Z

README_fuzz_testing.md

+- you manually kill the process e.g. (ctrl-C).
+
+Once you have built the fuzz testing suite using the commands listed above you
+can list the fuzz testing harnesses using the command;


grammar nit: : rather than ;

steven-johnson · 2023-05-15T17:01:08Z

README_fuzz_testing.md

+
+`crash-<some_random_hash>`
+
+To reproduce a crase we simply rerun our fuzz harness with our crash


steven-johnson · 2023-05-15T17:01:27Z

README_fuzz_testing.md

+reproduce the original crash.
+
+## Adding new fuzz tests
+A bare-bones fuzzer will look something like follows;


something like the following:

steven-johnson · 2023-05-15T17:31:22Z

README_fuzz_testing.md

+to do this we make use of a fuzzing-specific-toolchain/preset. e.g.
+
+```
+cmake . --preset linux-x64-fuzzer


This doesn't work as-is; e.g. LLVM_ROOT must be specified. For me, a complete commandline requires -DLLVM_ROOT=/path/to/llvminstall

Added that to the example

README_fuzz_testing.md

alexreinking

Some additional grammar changes. Also curious about the toolchain file... shouldn't it be included in the fuzzing preset?

README_fuzz_testing.md

cmake/toolchain.linux-x64-fuzzer.cmake

Co-authored-by: Alex Reinking <alex.reinking@gmail.com>

nathaniel-brough

Sorry for the slow response (and my terrible grammar :P)

There seems to still be some reproducibility issues caused by differing systems. I'm going to try going through these steps on a clean ubuntu install (probably in a docker container), and then use that as a bit of a starting point.

nathaniel-brough · 2023-05-20T20:44:52Z

README_fuzz_testing.md

+Halide has a set of fuzz-testing harnesses that can be used to find those
+tricky to find, edge cases and bugs that would otherwise not be caught
+by a regular unit-testing suite. At the moment these fuzz-tests are housed
+in the `//test/fuzz` directory. The fuzz testing suite use the common,


nathaniel-brough · 2023-05-20T20:44:56Z

README_fuzz_testing.md

+[libfuzzer](https://www.llvm.org/docs/LibFuzzer.html) interface for fuzz-tests.
+
+## Building fuzz tests
+Fuzz testing requires specific instrumentation across the entire build,


nathaniel-brough · 2023-05-20T20:45:14Z

README_fuzz_testing.md

+to do this we make use of a fuzzing-specific-toolchain/preset. e.g.
+
+```
+cmake . --preset linux-x64-fuzzer


Added that to the example

README_fuzz_testing.md

nathaniel-brough · 2023-05-20T20:52:22Z

README_fuzz_testing.md

+your disk.
+
+Up until this point the fuzzer has only been running on a single core. To
+speed things up a little, let's run the fuzzer in parrelell across all


nathaniel-brough · 2023-05-20T20:53:02Z

README_fuzz_testing.md

+
+`crash-<some_random_hash>`
+
+To reproduce a crase we simply rerun our fuzz harness with our crash


nathaniel-brough · 2023-05-20T20:53:15Z

README_fuzz_testing.md

+reproduce the original crash.
+
+## Adding new fuzz tests
+A bare-bones fuzzer will look something like follows;


cmake/toolchain.linux-x64-fuzzer.cmake

nathaniel-brough · 2023-05-20T22:51:58Z

Ok so after the latest update I was able to build successfully using the following Dockerfile;

FROM ubuntu:22.04

RUN apt-get update && apt-get install -y ninja-build lsb-core software-properties-common zlib1g-dev git binutils python3 python3-pip wget make
RUN pip3 install -U cmake
RUN wget https://apt.llvm.org/llvm.sh
RUN chmod +x llvm.sh
RUN ./llvm.sh 16
RUN apt-get install -y libclang-common-16-dev libclang-16-dev libclang1-16 clangd-16
RUN git clone --branch add_presets https://github.com/silvergasp/Halide.git
WORKDIR Halide
RUN cmake -B build --preset linux-x64-fuzzer -DLLVM_ROOT=/usr/lib/llvm-16
RUN cmake --build ./build -j$(nproc)
RUN ./build/test/fuzz/fuzz_simplify -help=1

Ideally we won't require docker to reproduce the build (otherwise we'd just use the google/oss-fuzz images), this was mostly just an exercise in capturing the parts of my system that might be different from others.

steven-johnson · 2023-05-31T18:58:40Z

Thanks for the update! This now works fine on my local Linux box, without any drama -- it should be easy for me to add to the buildbots after this lands.

I took the liberty of pushing a change to allow for the fuzz tests to run for a finite amount of time (100 runs per test) when used in combination with CTest. @alexreinking, PTAL, I had to tweak add_halide_test and friends a bit.

steven-johnson

LGTM, but I'd like @alexreinking approval too

alexreinking · 2023-05-31T19:31:44Z

I took the liberty of pushing a change to allow for the fuzz tests to run for a finite amount of time (100 runs per test) when used in combination with CTest. @alexreinking, PTAL, I had to tweak add_halide_test and friends a bit.

LGTM here

abadams · 2023-05-31T20:23:38Z

Does this also need a special llvm build? When I build LLVM using the instructions in the top-level readme, I get:

$ cmake -B fuzz_build --preset linux-x64-fuzzer -DLLVM_ROOT=/home/abadams/projects/llvm-main/install
Preset CMake variables:

  BUILD_SHARED_LIBS="NO"
  CMAKE_BUILD_TYPE="RelWithDebInfo"
  CMAKE_INSTALL_PREFIX:PATH="/home/abadams/projects/Halide/install/linux-x64-fuzzer"
  CMAKE_TOOLCHAIN_FILE:FILEPATH="/home/abadams/projects/Halide/cmake/toolchain.linux-x64-fuzzer.cmake"
  TARGET_WEBASSEMBLY="NO"
  WITH_PYTHON_BINDINGS="NO"
  WITH_TESTS="YES"
  WITH_TEST_AUTO_SCHEDULE="NO"
  WITH_TEST_CORRECTNESS="NO"
  WITH_TEST_ERROR="NO"
  WITH_TEST_FUZZ="YES"
  WITH_TEST_GENERATOR="NO"
  WITH_TEST_PERFORMANCE="NO"
  WITH_TEST_RUNTIME="NO"
  WITH_TEST_WARNING="NO"
  WITH_TUTORIALS="NO"
  WITH_UTILS="NO"

-- The C compiler identification is Clang 17.0.0
-- The CXX compiler identification is Clang 17.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /home/abadams/projects/llvm-main/install/bin/clang
-- Check for working C compiler: /home/abadams/projects/llvm-main/install/bin/clang - broken
CMake Error at /home/abadams/.local/lib/python3.10/site-packages/cmake/data/share/cmake-3.26/Modules/CMakeTestCCompiler.cmake:67 (message):
  The C compiler

    "/home/abadams/projects/llvm-main/install/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /home/abadams/projects/Halide/fuzz_build/CMakeFiles/CMakeScratch/TryCompile-SZttMN
    
    Run Build Command(s):/home/abadams/.local/lib/python3.10/site-packages/cmake/data/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile cmTC_13439/fast && /usr/bin/gmake  -f CMakeFiles/cmTC_13439.dir/build.make CMakeFiles/cmTC_13439.dir/build
    gmake[1]: Entering directory '/home/abadams/projects/Halide/fuzz_build/CMakeFiles/CMakeScratch/TryCompile-SZttMN'
    Building C object CMakeFiles/cmTC_13439.dir/testCCompiler.c.o
    /home/abadams/projects/llvm-main/install/bin/clang   -fsanitize=fuzzer-no-link  -MD -MT CMakeFiles/cmTC_13439.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_13439.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_13439.dir/testCCompiler.c.o -c /home/abadams/projects/Halide/fuzz_build/CMakeFiles/CMakeScratch/TryCompile-SZttMN/testCCompiler.c
    Linking C executable cmTC_13439
    /home/abadams/.local/lib/python3.10/site-packages/cmake/data/bin/cmake -E cmake_link_script CMakeFiles/cmTC_13439.dir/link.txt --verbose=1
    /home/abadams/projects/llvm-main/install/bin/clang -fsanitize=fuzzer-no-link  -fuse-ld=/bin/ld.lld  CMakeFiles/cmTC_13439.dir/testCCompiler.c.o -o cmTC_13439 
    ld.lld: error: cannot open /home/abadams/projects/llvm-main/install/lib/clang/17/lib/linux/libclang_rt.ubsan_standalone-x86_64.a: No such file or directory
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    gmake[1]: *** [CMakeFiles/cmTC_13439.dir/build.make:100: cmTC_13439] Error 1
    gmake[1]: Leaving directory '/home/abadams/projects/Halide/fuzz_build/CMakeFiles/CMakeScratch/TryCompile-SZttMN'
    gmake: *** [Makefile:127: cmTC_13439/fast] Error 2

abadams · 2023-05-31T21:13:55Z

Looks like we might need to add compiler-rt to the llvm projects list in the top-level readme

steven-johnson · 2023-05-31T21:24:54Z

Looks like we might need to add compiler-rt to the llvm projects list in the top-level readme

Ah, yes, I'll make that change. (My local LLVM builds have included that for a while because it's needed for other sanitizers)

EDIT: though, this means that installing a prebuilt LLVM means that builds will default to failing... we should probably try to smarten the CMake build to see if we can detect that compiler-rt is missing and just skip the fuzz tests instead. Let me see if I can manage that.

alexreinking · 2023-05-31T22:55:21Z

we should probably try to smarten the CMake build to see if we can detect that compiler-rt is missing and just skip the fuzz tests instead. Let me see if I can manage that.

try_compile with the fuzz flags should suffice

steven-johnson · 2023-05-31T23:08:12Z

try_compile with the fuzz flags should suffice

Yeah, and even that isn't necessary, since the fuzz tests are already elided if you aren't building with the toolchain file. I just updated the README files, I think that should suffice PTAL

* Adds fuzzing preset Partial fix for halide#7552 * Adds documentation on fuzz testing Closes: halide#7552 * Fixes spelling/grammar in fuzzing readme Co-authored-by: Alex Reinking <alex.reinking@gmail.com> * Remove asan flags from fuzzer * Add build directory in cmake/fuzzing documentation * Configure the fuzz tests to run for a finite amount of time * Update README * Update README_fuzz_testing.md * trigger buildbots * trigger buildbots * trigger buildbots * Update CMakeLists.txt --------- Co-authored-by: Steven Johnson <srj@google.com> Co-authored-by: Alex Reinking <alex.reinking@gmail.com>

Adds fuzzing preset

a31bbe3

Partial fix for halide#7552

nathaniel-brough force-pushed the add_presets branch from 2ea3475 to a31bbe3 Compare May 12, 2023 23:12

nathaniel-brough marked this pull request as ready for review May 12, 2023 23:12

nathaniel-brough force-pushed the add_presets branch from 9287bd7 to 29e45f1 Compare May 13, 2023 00:17

Adds documentation on fuzz testing

ae7a5bd

Closes: halide#7552

nathaniel-brough force-pushed the add_presets branch from 29e45f1 to ae7a5bd Compare May 13, 2023 00:18

nathaniel-brough mentioned this pull request May 13, 2023

Fuzzer tests need instructions on local reproduction #7552

Closed

steven-johnson requested review from alexreinking and halidebuildbots May 15, 2023 16:48

steven-johnson requested changes May 15, 2023

View reviewed changes

alexreinking requested changes May 17, 2023

View reviewed changes

steven-johnson mentioned this pull request May 17, 2023

UBSan failure reported from fuzz/simplify #7575

Closed

Merge branch 'main' into pr/7566

552bbe2

steven-johnson mentioned this pull request May 18, 2023

Simplification failure in fuzz/simplify #7584

Closed

Fixes spelling/grammar in fuzzing readme

ac0de71

Co-authored-by: Alex Reinking <alex.reinking@gmail.com>

nathaniel-brough force-pushed the add_presets branch from 648beb8 to ac0de71 Compare May 20, 2023 21:01

nathaniel-brough commented May 20, 2023

View reviewed changes

Remove asan flags from fuzzer

9c50965

Add build directory in cmake/fuzzing documentation

e892e7b

nathaniel-brough requested review from alexreinking and steven-johnson May 20, 2023 22:55

steven-johnson added 2 commits May 31, 2023 10:16

Merge branch 'main' into pr/7566

e5d5c93

Configure the fuzz tests to run for a finite amount of time

35740c5

steven-johnson approved these changes May 31, 2023

View reviewed changes

alexreinking approved these changes May 31, 2023

View reviewed changes

Update README

d05f0f3

steven-johnson added the release_notes For changes that may warrant a note in README for official releases. label May 31, 2023

steven-johnson added 5 commits May 31, 2023 16:23

Update README_fuzz_testing.md

0aa7523

trigger buildbots

d862d5b

trigger buildbots

4c9caf2

trigger buildbots

ba82811

Update CMakeLists.txt

880fad0

steven-johnson merged commit f3e1829 into halide:main Jun 1, 2023


		`crash-<some_random_hash>`

		To reproduce a crase we simply rerun our fuzz harness with our crash

Adds fuzzing preset #7566

Adds fuzzing preset #7566

Uh oh!

Conversation

nathaniel-brough commented May 12, 2023

Uh oh!

nathaniel-brough commented May 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steven-johnson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alexreinking left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nathaniel-brough left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nathaniel-brough commented May 20, 2023

Uh oh!

steven-johnson commented May 31, 2023

Uh oh!

steven-johnson left a comment

Choose a reason for hiding this comment

Uh oh!

alexreinking commented May 31, 2023

Uh oh!

abadams commented May 31, 2023

Uh oh!

nathaniel-brough commented May 12, 2023 •

edited

Loading

steven-johnson commented May 31, 2023 •

edited

Loading