Skip to content

fix(test/e2e): rewrite assert_events to avoid ending too soon sinsp-example log matching #2395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 7, 2025
Merged

fix(test/e2e): rewrite assert_events to avoid ending too soon sinsp-example log matching #2395

merged 1 commit into from
May 7, 2025

Conversation

FedeDP
Copy link
Contributor

@FedeDP FedeDP commented May 7, 2025

What type of PR is this?

/kind bug

Any specific area of the project related to this PR?

/area tests

Does this PR require a change in the driver versions?

What this PR does / why we need it:

Finally fixes e2e tests CI flakiness :)
Already restarted the CI many times; all failures i see come from container plugin 0.2.1 bug with podman, that will be solved with 0.2.2 (#2394); see CI runs: https://github.com/falcosecurity/libs/actions/runs/14865106080

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

@FedeDP @ekoops
fix(test/e2e): rewrite assert_events to avoid ending too soon sinsp…
148332d 
…-example log matching.

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

Co-authored-by: Leonardo Di Giovanna <leonardodigiovanna1@gmail.com>
@FedeDP
Copy link
Contributor Author

FedeDP commented May 7, 2025

/milestone 0.21.0

@poiana poiana added this to the 0.21.0 milestone May 7, 2025
@poiana poiana added the size/S label May 7, 2025
@poiana
Copy link
Contributor

poiana commented May 7, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: FedeDP

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@poiana poiana requested review from hbrueckner and Molter73 May 7, 2025 06:40
@poiana poiana added the approved label May 7, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented May 7, 2025

Perf diff from master - unit tests

     5.90%     -0.76%  [.] sinsp_parser::reset
     1.14%     +0.35%  [.] next
     2.34%     -0.26%  [.] sinsp_thread_manager::get_thread_ref
     0.67%     +0.25%  [.] sinsp_parser::event_cleanup
     1.51%     +0.24%  [.] is_conversion_needed
     3.74%     -0.22%  [.] next_event_from_file
     4.06%     +0.22%  [.] sinsp_thread_manager::find_thread
     0.80%     +0.18%  [.] sinsp_evt_filter::sinsp_evt_filter
     1.30%     -0.18%  [.] scap_event_decode_params
     2.59%     -0.18%  [.] gzfile_read

Heap diff from master - unit tests

peak heap memory consumption: 0B
peak RSS (including heaptrack overhead): 0B
total memory leaked: 0B

Heap diff from master - scap file

peak heap memory consumption: 0B
peak RSS (including heaptrack overhead): 0B
total memory leaked: 0B

Benchmarks diff from master

Comparing gbench_data.json to /root/actions-runner/_work/libs/libs/build/gbench_data.json
Benchmark                                                         Time             CPU      Time Old      Time New       CPU Old       CPU New
----------------------------------------------------------------------------------------------------------------------------------------------
BM_sinsp_split_mean                                            -0.0055         -0.0054           150           149           150           149
BM_sinsp_split_median                                          -0.0072         -0.0072           151           149           150           149
BM_sinsp_split_stddev                                          -0.1493         -0.1475             1             1             1             1
BM_sinsp_split_cv                                              -0.1446         -0.1428             0             0             0             0
BM_sinsp_concatenate_paths_relative_path_mean                  -0.0229         -0.0229            58            56            58            56
BM_sinsp_concatenate_paths_relative_path_median                -0.0242         -0.0242            58            56            58            56
BM_sinsp_concatenate_paths_relative_path_stddev                -0.6053         -0.6053             2             1             2             1
BM_sinsp_concatenate_paths_relative_path_cv                    -0.5960         -0.5960             0             0             0             0
BM_sinsp_concatenate_paths_empty_path_mean                     -0.0487         -0.0487            25            24            25            24
BM_sinsp_concatenate_paths_empty_path_median                   -0.0415         -0.0415            25            24            25            24
BM_sinsp_concatenate_paths_empty_path_stddev                   -0.8792         -0.8791             0             0             0             0
BM_sinsp_concatenate_paths_empty_path_cv                       -0.8730         -0.8729             0             0             0             0
BM_sinsp_concatenate_paths_absolute_path_mean                  +0.0556         +0.0556            56            59            56            59
BM_sinsp_concatenate_paths_absolute_path_median                +0.0650         +0.0650            56            59            56            59
BM_sinsp_concatenate_paths_absolute_path_stddev                +1.3323         +1.3334             1             2             1             2
BM_sinsp_concatenate_paths_absolute_path_cv                    +1.2095         +1.2105             0             0             0             0

@codecov Codecov
Copy link

codecov bot commented May 7, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.18%. Comparing base (bca865f) to head (148332d).
Report is 7 commits behind head on master.

Additional details and impacted files 
@@           Coverage Diff           @@
##           master    #2395   +/-   ##
=======================================
  Coverage   77.18%   77.18%           
=======================================
  Files         231      231           
  Lines       30361    30368    +7     
  Branches     4658     4657    -1     
=======================================
+ Hits        23434    23441    +7     
  Misses       6927     6927
Flag Coverage Δ
libsinsp 77.18% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

FedeDP
FedeDP commented May 7, 2025
View reviewed changes
test/e2e/tests/commons/sinspqa/sinsp.py
for event in expected_events:
success = False
received_event = None
start = datetime.now()
Copy link
Contributor Author

@FedeDP FedeDP May 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Idea is:

  • make use of timeout
  • for each log line, try to match it against all expected events
  • every time an event gets matched, remove it from the expected_events list
  • if we reach the timeout before matching all events, throw an assert failure

Previously, the code was error prone:

  • imagine len(expected_events) == 2
  • imagine sinsp.read() returned 50 lines
  • we matched expected_events[0]
  • but in the remaining read lines there was also expected_events[1]
  • we did not try to match it against all the remaining lines, basically throwing them in the trash
  • next for iteration, we try to match expected_events[1] against a new read of sinsp-example output that won't contain the requested event (because we already consumed the log lines where the event was present)

@poiana poiana added the lgtm label May 7, 2025
@github-project-automation github-project-automation bot moved this from Todo to In progress in Falco Roadmap May 7, 2025
@poiana poiana merged commit 8d44eca into master May 7, 2025
64 of 66 checks passed
@poiana poiana deleted the fix/e2e_tests_assert_events branch May 7, 2025 08:59
@github-project-automation github-project-automation bot moved this from In progress to Done in Falco Roadmap May 7, 2025
@bevanjkay bevanjkay mentioned this pull request May 20, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leogr leogr leogr approved these changes

@LucaGuerra LucaGuerra LucaGuerra approved these changes

@hbrueckner hbrueckner Awaiting requested review from hbrueckner

@Molter73 Molter73 Awaiting requested review from Molter73

Assignees

@leogr leogr

@LucaGuerra LucaGuerra

Labels
approved area/tests dco-signoff: yes kind/bug Something isn't working lgtm release-note-none size/S
Projects
Falco Roadmap
Status: Done
Milestone
0.21.0
Development

Successfully merging this pull request may close these issues.
4 participants
@FedeDP @poiana @leogr @LucaGuerra