Make fuzzing infrastructure more reliable #41288
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
03c058b to
bd01258
Compare
2330dfc to
6dc1a2d
Compare
|
/test |
6dc1a2d to
4d3812b
Compare
|
/test |
This fuzzer failed almost immediately with two panics for different
reasons, as seen below. Fix it.
When running the fuzzer via oss-fuzz, the framework replaces stdlib
testing with github.com/AdamKorcz/go-118-fuzz-build/testing which does
not currently support the entire Go 1.25 interface as expected by the
inner logic here. That causes some issues when attempting to compile the
fuzzer with that framework.
However, the good thing is that this fuzzer does not actually depend on
Hive, so the code doesn't need to pass a hive typed logger. Instead,
fall back to a standard logger with no output (since the fuzzer will be
putting all sorts of weird and wonderful input into this path anyway).
$ go test ./pkg/fqdn/namemanager/ -fuzz=FuzzMapSelectorsToNamesLocked -fuzztime=5s
cilium.io
fuzz: elapsed: 0s, gathering baseline coverage: 0/1 completed
failure while testing seed corpus entry: FuzzMapSelectorsToNamesLocked/582528ddfad69eb5
fuzz: elapsed: 0s, gathering baseline coverage: 0/1 completed
--- FAIL: FuzzMapSelectorsToNamesLocked (0.11s)
--- FAIL: FuzzMapSelectorsToNamesLocked (0.00s)
testing.go:1693: panic: reflect: call of reflect.Value.Elem on struct Value
goroutine 73 [running]:
runtime/debug.Stack()
/snap/go/10927/src/runtime/debug/stack.go:26 +0x9b
testing.tRunner.func1()
/snap/go/10927/src/testing/testing.go:1693 +0x1c8
panic({0x45cb4e0?, 0xc000c80078?})
/snap/go/10927/src/runtime/panic.go:792 +0x132
reflect.Value.Elem({0x47fa220?, 0xc000bb6100?, 0x4ada380?})
/snap/go/10927/src/reflect/value.go:1265 +0x190
github.com/AdaLogics/go-fuzz-headers.(*ConsumeFuzzer).GenerateStruct(0xc0004c41c0, {0x47fa220?, 0xc000bb6100?})
/var/git/cilium/vendor/github.com/AdaLogics/go-fuzz-headers/consumer.go:117 +0x185
github.com/cilium/cilium/pkg/fqdn/namemanager.FuzzMapSelectorsToNamesLocked.func1(0x0?, {0xc000440170, 0x1, 0x8})
/var/git/cilium/pkg/fqdn/namemanager/fuzz_test.go:20 +0x11f
reflect.Value.call({0x44be4e0?, 0xc00045b670?, 0x13?}, {0x4c5a023, 0x4}, {0xc000bb4db0, 0x2, 0x2?})
/snap/go/10927/src/reflect/value.go:584 +0xca6
reflect.Value.Call({0x44be4e0?, 0xc00045b670?, 0xc0007be000?}, {0xc000bb4db0?, 0x4c531c0?, 0x522d7fa?})
/snap/go/10927/src/reflect/value.go:368 +0xb9
testing.(*F).Fuzz.func1.1(0xc000802380?)
/snap/go/10927/src/testing/fuzz.go:340 +0x2f5
testing.tRunner(0xc000802380, 0xc0005fe000)
/snap/go/10927/src/testing/testing.go:1792 +0xf4
created by testing.(*F).Fuzz.func1 in goroutine 23
/snap/go/10927/src/testing/fuzz.go:327 +0x5fa
FAIL
exit status 1
FAIL github.com/cilium/cilium/pkg/fqdn/namemanager 0.150s
$ go test ./pkg/fqdn/namemanager/ -fuzz=FuzzMapSelectorsToNamesLocked -fuzztime=5s
cilium.io
fuzz: elapsed: 0s, gathering baseline coverage: 0/1 completed
failure while testing seed corpus entry: FuzzMapSelectorsToNamesLocked/582528ddfad69eb5
fuzz: elapsed: 0s, gathering baseline coverage: 0/1 completed
--- FAIL: FuzzMapSelectorsToNamesLocked (0.13s)
--- FAIL: FuzzMapSelectorsToNamesLocked (0.00s)
testing.go:1693: panic: testing: f.Cleanup was called inside the fuzz target, use t.Cleanup instead
goroutine 60 [running]:
runtime/debug.Stack()
/snap/go/10927/src/runtime/debug/stack.go:26 +0x9b
testing.tRunner.func1()
/snap/go/10927/src/testing/testing.go:1693 +0x1c8
panic({0x4427600?, 0xc0004f4250?})
/snap/go/10927/src/runtime/panic.go:792 +0x132
testing.(*common).checkFuzzFn(...)
/snap/go/10927/src/testing/testing.go:720
testing.(*common).Cleanup(0x47cf00?, 0x200000c000ba36b8?)
/snap/go/10927/src/testing/testing.go:1191 +0x26d
github.com/cilium/hive/hivetest.Logger({0x5151a60, 0xc000971600}, {0x0, 0x0, 0x4?})
/var/git/cilium/vendor/github.com/cilium/hive/hivetest/slog.go:19 +0xa4
github.com/cilium/cilium/pkg/fqdn/namemanager.FuzzMapSelectorsToNamesLocked.func1(0x0?, {0xc000492180, 0x1, 0x8})
/var/git/cilium/pkg/fqdn/namemanager/fuzz_test.go:22 +0x128
reflect.Value.call({0x44be4e0?, 0xc0004f4e00?, 0x13?}, {0x4c5a023, 0x4}, {0xc000b92270, 0x2, 0x2?})
/snap/go/10927/src/reflect/value.go:584 +0xca6
reflect.Value.Call({0x44be4e0?, 0xc0004f4e00?, 0xc000750000?}, {0xc000b92270?, 0x4c531c0?, 0x522d7da?})
/snap/go/10927/src/reflect/value.go:368 +0xb9
testing.(*F).Fuzz.func1.1(0xc000c008c0?)
/snap/go/10927/src/testing/fuzz.go:340 +0x2f5
testing.tRunner(0xc000c008c0, 0xc000744000)
/snap/go/10927/src/testing/testing.go:1792 +0xf4
created by testing.(*F).Fuzz.func1 in goroutine 35
/snap/go/10927/src/testing/fuzz.go:327 +0x5fa
FAIL
exit status 1
FAIL github.com/cilium/cilium/pkg/fqdn/namemanager 0.160s
Fixes: 44f7c87 ("pkg/fqdn: migrate to slog")
Signed-off-by: Joe Stringer <joe@cilium.io>
The fuzzer test workflows we have rely on the fuzzers being in a filename that ends with 'fuzz_test.go'. Move the fuzzer there to make sure that any changes to the fuzzer will trigger the test workflows. Signed-off-by: Joe Stringer <joe@cilium.io>
This file will be linked into a fuzzer, which requires the package to be identical to the functions that it tests. Fix up the package name. Signed-off-by: Joe Stringer <joe@cilium.io>
Add missing fuzzers, and sort them alphabetically. Some fuzzers rely on additional Go code, so ensure that the dependencies are also moved into a location where oss-fuzz can find them. Signed-off-by: Joe Stringer <joe@cilium.io>
Moving files around makes it more painful to locally build fuzzers and reproduce results reported by oss-fuzz, because the run is no longer idempotent. Instead of moving files around in the source tree, just link them so that oss-fuzz can find them. Tested by checking out the oss-fuzz repository, then: $ python infra/helper.py build_fuzzers cilium /path/to/cilium Signed-off-by: Joe Stringer <joe@cilium.io>
This should make the fuzzer invocations more consistent. Signed-off-by: Joe Stringer <joe@cilium.io>
Rather than writing an extra random file to the repository, just add github.com/AdamKorcz/go-118-fuzz-build/testing as a 'tool' dependency so that the underlying library is always available in the repository for use by fuzzers. Signed-off-by: Joe Stringer <joe@cilium.io>
The purpose of this script is to ensure that all fuzzers in the tree are also added into the oss-fuzz scripting for longer term fuzzing. Signed-off-by: Joe Stringer <joe@cilium.io>
Add 'make fuzz' target which can be run directly from the top of the tree in order to run the fuzzers that are checked in. Ideally this would just reuse the same infrastructure as oss-fuzz, but oss-fuzz adds a lot of its own complexity, it's slower, and it's further from how Go tooling would natively run a fuzzer. Due to the lower complexity, this script has a lower likelihood of breaking over time. Signed-off-by: Joe Stringer <joe@cilium.io>
4d3812b to
a64c623
Compare
|
/test EDIT: CI-Fuzz workflow is happy: https://github.com/cilium/cilium/actions/runs/17132259586/job/48599343545?pr=41288 |
aanm
approved these changes
Aug 22, 2025
christarazi
approved these changes
Aug 22, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Review commit by commit.
Fix some inconsistencies in the fuzzing infrastructure we have, and apply new
checks / expectations for fuzzers that live in the main branch. Fix one of the
broken fuzzers,
FuzzMapSelectorsToNamesLocked.For simplicity of reproducing generic fuzzer errors, add a new 'make fuzz'
target, which optionally accepts a
FUZZ_TIMEvariable to provide a targettime to run each fuzzers in the tree. The full run time will compile the
fuzzers and then run each fuzzer for this time period. Currently this appears
to take a minute or so, but will take a little longer when the full set of
fuzzers from github.com/cncf/cncf-fuzzing are integrated into this tree.
After this, the CIFuzz workflow should now run some of the newer fuzzers
which were introduced into Cilium more recently.
If developers add new fuzzers to the tree, a 'check-fuzz' target will run and
inform them about how to properly format that change to ensure that the fuzzer
is integrated into the fuzz testing that the Cilium project performs.
Note that most of the LoC churn here is adding a dependency that is currently
resolved dynamically as part of the oss-fuzz build: