halide: initial integration #10107

nathaniel-brough · 2023-04-17T19:19:09Z

No description provided.

nathaniel-brough · 2023-04-17T19:30:49Z

TODOs:

Add primary contact emails
Switch from my personal branch back to the mainline Halide branch
Add CC contact emails

nathaniel-brough · 2023-04-17T19:34:46Z

Halide is critical to the global IT infrastructure and has;

5.4k stars on github
An OSSF criticality score of 0.702
Has already been approached by the 'google-autofuzz' bot.

CC @steven-johnson @alexreinking

projects/halide/build.sh

nathaniel-brough · 2023-05-01T17:07:39Z

projects/halide/project.yaml

+homepage: "https://halide-lang.org/"
+language: c++
+# TODO: Find primary contact
+primary_contact: ""


@alexreinking would it make sense for you to be the primary contact for this repo? if so do you have a google/gmail or github associated email that I can put here?

Probably better to put @steven-johnson as OSS-Fuzz is a Google project and he works at Google.

If you would like I can add you to the auto_ccs instead and that'll give you access to the dashboard/stats/bug tracker etc. I think the primary contact has a few more privileges, on top of that but I'm unsure.

Probably better to put @steven-johnson as OSS-Fuzz is a Google project and he works at Google.

SGTM

If you would like I can add you to the auto_ccs instead and that'll give you access to the dashboard/stats/bug tracker etc. I think the primary contact has a few more privileges, on top of that but I'm unsure.

That would be good!

Easy, should I use your @gmail from the git log or a different email. Note that it does need to be a gmail account for you to get full access to the bug tracker/dashboard.

Yeah, alex.reinking@gmail.com is the best address

nathaniel-brough · 2023-05-01T23:07:57Z

Build failures should be fixed in halide/Halide#7542

nathaniel-brough · 2023-05-01T23:11:07Z

I'm just confirming the change by creating a PR against silvergasp/main and cloning from that rather than upstream halide. CI results here nathaniel-brough#8

jonathanmetzman · 2023-05-04T13:52:47Z

Is this ready to merge. We approve adding it to OSS-Fuzz.

steven-johnson · 2023-05-04T16:49:56Z

Is this ready to merge. We approve adding it to OSS-Fuzz.

Since half the tests seem to be failing, I'm gonna guess "no"

nathaniel-brough · 2023-05-04T16:56:57Z

Yeah, this not quite ready. Although I think I just need to re-trigger the CI as I fixed some issues in the CmakeLists upstream. I'll try the close/re-open hack to re-trigger.

nathaniel-brough · 2023-05-04T17:07:43Z

While I've got your attention @jonathanmetzman have you ever seen an error like this? It almost looks like a bug in libfuzzer, the stack trace comes from _start and seems to be related to global c++ constructors so the bad_build check is failing before it's even getting to main.

I haven't seen this problem while building the fuzzers outside of the oss-fuzz container with the same build commands, and the fuzzers runs for at least a couple hours (before I've manually killed them) while I've been testing.

https://github.com/silvergasp/oss-fuzz/actions/runs/4855944456/jobs/8654990302?pr=8

ERROR: AddressSanitizer: container-overflow on address 0x6080000005e8 at pc 0x00000057c921 bp 0x7ffddbfa0c50 sp 0x7ffddbfa0420
READ of size 24 at 0x6080000005e8 thread T0
SCARINESS: 26 (multi-byte-read-container-overflow)
    #0 0x57c920 in __asan_memcpy /src/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:22:3
    #1 0x670324 in basic_string /usr/local/bin/../include/c++/v1/string:1950:7
    #2 0x670324 in construct<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > /usr/local/bin/../include/c++/v1/__memory/allocator.h:151:28
    #3 0x670324 in construct<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void> /usr/local/bin/../include/c++/v1/__memory/allocator_traits.h:290:13
    #4 0x670324 in __construct_backward_with_exception_guarantees<std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > *> /usr/local/bin/../include/c++/v1/memory:921:9
    #5 0x670324 in std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >::__swap_out_circular_buffer(std::__1::__split_buffer<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&>&) /usr/local/bin/../include/c++/v1/vector:892:5
    #6 0x66fec3 in void std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >::__push_back_slow_path<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /usr/local/bin/../include/c++/v1/vector:1530:5
    #7 0x2d275be in push_back /usr/local/bin/../include/c++/v1/vector:1543:9
    #8 0x2d275be in insert /src/llvm-project/llvm/include/llvm/ADT/UniqueVector.h:51:12
    #9 0x2d275be in llvm::DebugCounter::addCounter(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /src/llvm-project/llvm/include/llvm/Support/DebugCounter.h:161:42
    #10 0x2d0916e in llvm::DebugCounter::registerCounter(llvm::StringRef, llvm::StringRef) /src/llvm-project/llvm/include/llvm/Support/DebugCounter.h:70:23
    #11 0x45cb50 in __cxx_global_var_init /src/llvm-project/llvm/lib/Transforms/Scalar/DCE.cpp:36:1
    #12 0x45cb50 in _GLOBAL__sub_I_DCE.cpp /src/llvm-project/llvm/lib/Transforms/Scalar/DCE.cpp
    #13 0x4f1ccbc in __libc_csu_init (/tmp/not-out/tmprf7u3gw_/fuzz_simplify+0x4f1ccbc)
    #14 0x7f946b06800f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2400f) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
    #15 0x46dd2d in _start (/tmp/not-out/tmprf7u3gw_/fuzz_simplify+0x46dd2d)

DEDUP_TOKEN: __asan_memcpy--basic_string--construct<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >
0x6080000005e8 is located 72 bytes inside of 96-byte region [0x6080000005a0,0x608000000600)
allocated by thread T0 here:
    #0 0x5b83dd in operator new(unsigned long) /src/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:95:3
    #1 0x66fe0c in __libcpp_operator_new<unsigned long> /usr/local/bin/../include/c++/v1/new:245:10
    #2 0x66fe0c in __libcpp_allocate /usr/local/bin/../include/c++/v1/new:271:10
    #3 0x66fe0c in allocate /usr/local/bin/../include/c++/v1/__memory/allocator.h:105:38
    #4 0x66fe0c in allocate /usr/local/bin/../include/c++/v1/__memory/allocator_traits.h:262:20
    #5 0x66fe0c in __split_buffer /usr/local/bin/../include/c++/v1/__split_buffer:306:29
    #6 0x66fe0c in void std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >::__push_back_slow_path<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /usr/local/bin/../include/c++/v1/vector:1526:49
    #7 0x2d275be in push_back /usr/local/bin/../include/c++/v1/vector:1543:9
    #8 0x2d275be in insert /src/llvm-project/llvm/include/llvm/ADT/UniqueVector.h:51:12
    #9 0x2d275be in llvm::DebugCounter::addCounter(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /src/llvm-project/llvm/include/llvm/Support/DebugCounter.h:161:42
    #10 0x2d0916e in llvm::DebugCounter::registerCounter(llvm::StringRef, llvm::StringRef) /src/llvm-project/llvm/include/llvm/Support/DebugCounter.h:70:23
    #11 0x45a075 in __cxx_global_var_init /src/llvm-project/llvm/lib/CodeGen/MachineCopyPropagation.cpp:83:1
    #12 0x45a075 in _GLOBAL__sub_I_MachineCopyPropagation.cpp /src/llvm-project/llvm/lib/CodeGen/MachineCopyPropagation.cpp
    #13 0x4f1ccbc in __libc_csu_init (/tmp/not-out/tmprf7u3gw_/fuzz_simplify+0x4f1ccbc)

DEDUP_TOKEN: operator new(unsigned long)--__libcpp_operator_new<unsigned long>--__libcpp_allocate
HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.
If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.
SUMMARY: AddressSanitizer: container-overflow /src/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:22:3 in __asan_memcpy
Shadow bytes around the buggy address:
  0x0c107fff8060: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
  0x0c107fff8070: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
  0x0c107fff8080: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
  0x0c107fff8090: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
  0x0c107fff80a0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
=>0x0c107fff80b0: fa fa fa fa 00 00 00 00 00 00 00 00 00[fc]fc fc
  0x0c107fff80c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c107fff80d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c107fff80e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c107fff80f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c107fff8100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb

steven-johnson · 2023-05-04T17:45:21Z

See also halide/Halide#7546, which must be fixed before this can land

nathaniel-brough · 2023-05-05T00:13:30Z

Ok so it looks like the bad-build failure might be a false positive bug in llvm/ASAN see llvm/llvm-project#60384 which looks similar. I'm not really that familiar with llvm internals so I'm not confident enough to say, "yes it is a bug" and that we should just disable container overflow checks for now... I'll investigate further, it's fairly easy to disable this ASAN check but not sure if that's the right approach.

nathaniel-brough · 2023-05-06T00:02:04Z

So after A LOT of fiddling/waiting 3 or so hours to rebuild I've finally got this working. It turns out ASAN has a bunch of false positives if you are linking non-ASAN-instrumented code with ASAN instrumented code. But LLVM doesn't necessarily like to build directly with the flags provided by the OSS-fuzz build environment, and it needs it's own cmake configuration to build libclang and libllvm with a sanitized build. It'll be another few hours until the CI is finished but I'm pretty confident that it'll all be green as I've tested it locally.

nathaniel-brough · 2023-05-06T04:31:33Z

@jonathanmetzman I think this one is ready to go, the build failure is because the github runner ran out of disk space while building LLVM and doesn't appear to be related to the fuzzer itself. As long as the machines that oss-fuzz uses for builds have the disk space to build everything, then this should be good to go. Otherwise the bad-build checks are all passing.

nathaniel-brough · 2023-05-20T23:04:08Z

@jonathanmetzman friendly ping

github-actions · 2023-06-12T18:35:19Z

@silvergasp is integrating a new project:
- Main repo: https://github.com/halide/Halide
- Criticality score: 0.69630

nathaniel-brough · 2023-09-29T01:36:18Z

@jonathanmetzman Friendly ping :) The CI is still failing because the github builder is running out of disk space while building llvm.

Signed-off-by: Nathaniel Brough <nathaniel.brough@gmail.com>

nathaniel-brough force-pushed the halide branch from 64bac4b to 7ebb13e Compare April 17, 2023 19:19

nathaniel-brough mentioned this pull request Apr 17, 2023

Add libfuzzer compatible fuzz harness halide/Halide#7512

Merged

alexreinking reviewed Apr 17, 2023

View reviewed changes

projects/halide/build.sh Show resolved Hide resolved

nathaniel-brough force-pushed the halide branch 4 times, most recently from 18dffe5 to 4eb7473 Compare April 18, 2023 22:47

nathaniel-brough force-pushed the halide branch from 4eb7473 to bacd7f5 Compare May 1, 2023 17:01

nathaniel-brough commented May 1, 2023

View reviewed changes

nathaniel-brough force-pushed the halide branch 2 times, most recently from 236a9b1 to 3b76ec4 Compare May 1, 2023 18:37

nathaniel-brough marked this pull request as ready for review May 1, 2023 18:38

nathaniel-brough force-pushed the halide branch from 3b76ec4 to 80ec59c Compare May 1, 2023 20:33

nathaniel-brough mentioned this pull request May 1, 2023

Fix flag check for fuzzers halide/Halide#7542

Merged

nathaniel-brough closed this May 4, 2023

nathaniel-brough reopened this May 4, 2023

nathaniel-brough force-pushed the halide branch from 80ec59c to 13202b3 Compare May 5, 2023 23:54

nathaniel-brough mentioned this pull request May 13, 2023

Fuzzer tests need instructions on local reproduction halide/Halide#7552

Closed

nathaniel-brough force-pushed the halide branch from 13202b3 to 9d2e765 Compare June 7, 2023 21:09

jonathanmetzman enabled auto-merge (squash) June 7, 2023 21:30

nathaniel-brough mentioned this pull request Jun 8, 2023

Can't build Halide/LLVM (No space left on device) #10491

Closed

nathaniel-brough closed this Jun 12, 2023

auto-merge was automatically disabled June 12, 2023 18:33
Pull request was closed

nathaniel-brough reopened this Jun 12, 2023

nathaniel-brough force-pushed the halide branch from 9d2e765 to 920a9da Compare June 14, 2023 23:32

nathaniel-brough force-pushed the halide branch from 920a9da to bfb1cd2 Compare December 4, 2023 02:34

halide: initial integration

672efad

Signed-off-by: Nathaniel Brough <nathaniel.brough@gmail.com>

nathaniel-brough force-pushed the halide branch from bfb1cd2 to 672efad Compare December 5, 2023 00:48

jonathanmetzman merged commit 6648c10 into google:master Dec 5, 2023

halide: initial integration #10107

halide: initial integration #10107

Uh oh!

Conversation

nathaniel-brough commented Apr 17, 2023

Uh oh!

nathaniel-brough commented Apr 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nathaniel-brough commented Apr 17, 2023

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nathaniel-brough commented May 1, 2023

Uh oh!

nathaniel-brough commented May 1, 2023

Uh oh!

jonathanmetzman commented May 4, 2023

Uh oh!

steven-johnson commented May 4, 2023

Uh oh!

nathaniel-brough commented May 4, 2023

Uh oh!

nathaniel-brough commented May 4, 2023

Uh oh!

steven-johnson commented May 4, 2023

Uh oh!

nathaniel-brough commented May 5, 2023

Uh oh!

nathaniel-brough commented May 6, 2023

Uh oh!

nathaniel-brough commented May 6, 2023

Uh oh!

nathaniel-brough commented May 20, 2023

Uh oh!

github-actions bot commented Jun 12, 2023

Uh oh!

nathaniel-brough commented Sep 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

nathaniel-brough commented Apr 17, 2023 •

edited

Loading

nathaniel-brough commented Sep 29, 2023 •

edited

Loading