workflows: tests: Fix arg to check marks #1064
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We used `-k "not benchmark"` which would filter out ANY test that contained the word 'benchmark'. This change fixes that by using the `-m` argument to only filter on marks.
1 similar comment
|
Demonstration of the issue: root@pwnvm:~/code/gef# pytest --collect-only -k "benchmark" tests/
=========================================== test session starts ============================================
platform linux -- Python 3.10.12, pytest-7.4.0, pluggy-1.2.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /root/code/gef/tests
configfile: pytest.ini
plugins: xdist-3.5.0, cov-4.1.0, benchmark-4.0.0
collected 172 items / 168 deselected / 4 selected
<Package api>
<Module deprecated.py>
<UnitTestCase GefFuncDeprecatedApi>
<TestCaseFunction test_benchmark>
<Package perf>
<Module benchmark.py>
<UnitTestCase BenchmarkBasicApi>
<TestCaseFunction test_cmd_context>
<TestCaseFunction test_elf_parsing>
<TestCaseFunction test_gef_memory_maps>
============================= 4/172 tests collected (168 deselected) in 0.33s ==============================We see with root@pwnvm:~/code/gef# pytest --collect-only -m "benchmark" tests/
=========================================== test session starts ============================================
platform linux -- Python 3.10.12, pytest-7.4.0, pluggy-1.2.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /root/code/gef/tests
configfile: pytest.ini
plugins: xdist-3.5.0, cov-4.1.0, benchmark-4.0.0
collected 172 items / 169 deselected / 3 selected
<Package perf>
<Module benchmark.py>
<UnitTestCase BenchmarkBasicApi>
<TestCaseFunction test_cmd_context>
<TestCaseFunction test_elf_parsing>
<TestCaseFunction test_gef_memory_maps>
============================= 3/172 tests collected (169 deselected) in 0.26s ==============================Here we don't find that test. In the workflow we use the opposite logic, so we would accidentally NOT run tests with that word. |
hugsy
approved these changes
Jan 30, 2024
hugsy
pushed a commit
to hugsy/gef-extras
that referenced
this pull request
Jan 30, 2024
We used `-k "not benchmark"` which would filter out ANY test that contained the word 'benchmark'. This change fixes that by using the `-m` argument to only filter on marks. Same change as hugsy/gef#1064
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We used
-k "not benchmark"which would filter out ANY test that contained the word 'benchmark'.This change fixes that by using the
-margument to only filter on marks.