-
Notifications
You must be signed in to change notification settings - Fork 3.4k
bpf: scapy support (dev. experience) #40294
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@cilium/sig-datapath looking for your feedback on this one. |
FWIW, I'll also provide my feedback.
I'll update my comment in case something else comes to my mind 🤔 |
@smagnani96 thx for the feedback, appreciated.
Thx. I think it can be polished slightly.
Actually, it might just work today with any sort of buf. The macro comparing already takes an offset, so that should work also out of the box. I will double-check.
Not sure I understood your comment; you would have a separate
That would be for v4 L2 announcement only (tc_l2_announcement.c)
Bumping We could could also only conditionally define |
Yep, I was referring more of when dumping the packet in expected = Ether(bytes.fromhex(data["expected"]))
got = Ether(bytes.fromhex(data["got"]))
Ha! Ok got it. I thought you wanted to make it even more granular, like specifying the buffer only locally to the
Or pass it as a const to the |
Yep, you are right, that's the part that is missing (I forgot). Let me think about options, but it might be as easy as to add the first expected Layer as part of the macro.
To be honest, today you can define them in
TODO3 suggests to create one of these python "envs" with the packet definitions per unit test so that there are no possible conflicts. It's really a detail, because we could also continue to use a single
Let me look into it. |
[...]
Overall I like the series! After the base is in, I presume other devs who want to use it won't have to touch the python parts, right? I'd think we should probably convert all the packet builders with this work to simplify the code. What would be great to improve imho is the macro namings, e.g. things like SCAPY_ASSERT_PKT_BUF, SCAPY_ASSERT_PKT_BUF_OFF, SCAPY_PKT_BUILDER.. maybe we can simplify the names a bit and add documentation for them with a small example. Right now one has to read the macros to understand what they are doing. Small nit also wrt API: SCAPY_ASSERT_PKT_BUF_OFF has |
Correct. The expectation is that no one has to write any Python code other than their packet definitions, e.g.:
Fully agree. Not good at names, so any suggestions welcome.
I will certainly do that if we are in agreement it makes sense to be merged.
Good catch. Totally unintentional, should be |
Another side note -- no need to address in the first PR. |
Commits 75ff49d, 8f3ab89 do not match "(?m)^Signed-off-by:". Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin |
Adding a few extra bits for future enhancements. $ for f in classifiers_l3_dev classifiers_l2_dev; do make -C bpf/tests $f.o && go test ./bpf/tests/bpftest -bpf-test-path $pwd/bpf/tests -exec "sudo -E" -test $f -v; done This will run only I think one of these two can be added in a 2nd moment (I can even do it, preferring 1):
|
This commit mounts /sys when invoking contrib/scripts/builder.sh, so that BPF unit tests with the scapy support can read from the trace_pipe. It also changes builder.sh to receive quoted DOCKER_ARGS. Signed-off-by: Marc Suñé <marc.sune@isovalent.com>
This commit refactors `tc_l2_announcement.c` to make use of the new scapy test infrastructure (showcase). Signed-off-by: Marc Suñé <marc.sune@isovalent.com>
Add SPDX Apache license identifier in scapy-related Python scripts. Signed-off-by: Simone Magnani <simone.magnani@isovalent.com> Signed-off-by: Marc Suñé <marc.sune@isovalent.com>
Add a small guide on how to write BPF unit tests with Scapy along with a short intro to Scapy. Signed-off-by: Marc Suñé <marc.sune@isovalent.com>
5a62c60
to
fa864d6
Compare
Ready to merge:
|
It looks like script install-builder-deps.sh has never been used for generating builder images. Ever since it was introduced in commit 7724ceb ("build: New builder image supporting cross-compilation"), it's never been called anywhere in the repository. Remove it to avoid confusion as in cilium#40838. Link: cilium#40294 (comment) Signed-off-by: Quentin Monnet <qmo@qmon.net>
/test Checkpatch's failure is due to a warning about the use of |
@smagnani96 I see you self-requested a review a few days ago, do you want to take another look or are you good with this PR? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like my own review was missing :)
I'm reviewing this right now, was waiting for the force-pushes 🙏🏼 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Would probably create (if not done already) an GH issue to track possible TODOs and enhancements to this (e.g., the cocci script).
Amazing Marc! 🚀
Will do once merged, thx! |
It looks like script install-builder-deps.sh has never been used for generating builder images. Ever since it was introduced in commit 7724ceb ("build: New builder image supporting cross-compilation"), it's never been called anywhere in the repository. Remove it to avoid confusion as in #40838. Link: #40294 (comment) Signed-off-by: Quentin Monnet <qmo@qmon.net>
Add python3-scapy and python3-jinja2 packages to install-deps script. This is requirement coming from the BPF unit test scapy support PR (cilium#40294). Signed-off-by: Marc Suñé <marc.sune@isovalent.com>
Commit bda1e68 added the two dependencies as described in the log in `images/builder/install-builder-deps.sh`. It turns out this file is not used (at all) and hence a separate PR has been opened to remove it entirely. > Add python3-scapy and python3-jinja2 packages to install-deps script. > This is requirement coming from the BPF unit test scapy support PR > (cilium#40294). This commit adds it as part of the builder `Dockerfile` so that scapy and jinja2 are.. there :) Signed-off-by: Marc Suñé <marc.sune@isovalent.com>
It looks like script install-builder-deps.sh has never been used for generating builder images. Ever since it was introduced in commit 7724ceb ("build: New builder image supporting cross-compilation"), it's never been called anywhere in the repository. Remove it to avoid confusion as in cilium#40838. Link: cilium#40294 (comment) Signed-off-by: Quentin Monnet <qmo@qmon.net>
This patchset adds scapy support in two different ways to (hopefully) improve developer experience when writing and debugging BPF code, and when writing BPF unit tests.
Note: this patchset does NOT introduce a new testing framework.
hexdump()
lib/hexdump.h
is a new header that defineshexdump()
andHEXDUMP()
primitives toprintk()
(up to) the first64128b bytes of a packet. It is meant to be used solely for debugging purposes of BPF code.tools/log2scapy.py
can then be used to parse thetrace_pipe
output and:.show()
routine: e.g.scapy
interactive shell with the parsed packets (e.g. to manipulate them or inject them somewhere)Short demo
hexdump_scapy2.mp4
Other comments:
hexdump()
/HEXDUMP()
is usedScapy in BPF unit tests
This patchset also adds scapy support to BPF unit tests. The support is additive, and can be used to:
BUF_DECL()
)You can see a test ported to scapy infra here
Short demo
bpf_unit_scapy_diff.mp4
Other comments
scapy
syntax.Final remarks
I'd like to get thoughts from the community on this to continue investing in it.
While we all get "used" to write BPF and BPF tests, IMO s/t like this would have saved me some precious dev time.
Fixed in latest commit.checkpatch.pl
errors: some style errors are a result of defining scapy pkts within BPF code. This could be sorted out by TODO3, which would mean moving the packet definition itself to a python file (that would be loaded byscapy/parser.py
). This might be cleaner.Something like: