--stepwise-skip has no effect without --stepwise #9062
Description
- a detailed description of the bug or problem you are having
The flag --stepwise will "exit on test failure and continue from last failing test".
The flag --stepwise-skip is supposed to "ignore the first failing test but stop on the next failing test".
However, it does nothing unless used in conjunction with --stepwise. The combo requirement is not mentioned in the help text, seems redundant, and is surprising behavior.
I recommend that --stepwise-skip should act the same as --stepwise --stepwise-skip.
- output of
pip listfrom the virtual environment you are using
$ pip list
Package Version
---------- -------
attrs 21.2.0
iniconfig 1.1.1
packaging 21.0
pip 21.2.4
pluggy 0.13.1
py 1.10.0
pyparsing 2.4.7
pytest 6.2.4
setuptools 46.4.0
toml 0.10.2
wheel 0.34.2- pytest and operating system versions
MacOS, Python 3.9.5, pytest-6.2.4
- minimal example if possible
Example test_step.py:
def test_one():
assert False
def test_two():
assert False
def test_three():
assert Falsethree failures:
$ pytest --tb=no test_step.py
========================= test session starts ==========================
platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/okken/projects/sw
collected 3 items
test_step.py FFF [100%]
======================= short test summary info ========================
FAILED test_step.py::test_one - assert False
FAILED test_step.py::test_two - assert False
FAILED test_step.py::test_three - assert False
========================== 3 failed in 0.01s ===========================
--stepwise-skip has no effect:
$ pytest --tb=no --stepwise-skip test_step.py
========================= test session starts ==========================
platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/okken/projects/sw
collected 3 items
test_step.py FFF [100%]
======================= short test summary info ========================
FAILED test_step.py::test_one - assert False
FAILED test_step.py::test_two - assert False
FAILED test_step.py::test_three - assert False
========================== 3 failed in 0.01s ===========================--stepwise works as expected, stopping after first failure:
$ pytest --tb=no --stepwise test_step.py
========================= test session starts ==========================
platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/okken/projects/sw
collected 3 items
stepwise: no previously failed tests, not skipping.
test_step.py F
======================= short test summary info ========================
FAILED test_step.py::test_one - assert False
!!!! Interrupted: Test failed, continuing from this test next run. !!!!!
========================== 1 failed in 0.07s ===========================--stepwise-skip only works with --stepwise to stop after second failure:
$ pytest --tb=no --stepwise --stepwise-skip test_step.py
========================= test session starts ==========================
platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/okken/projects/sw
collected 3 items
stepwise: skipping 0 already passed items.
test_step.py FF
======================= short test summary info ========================
FAILED test_step.py::test_one - assert False
FAILED test_step.py::test_two - assert False
!!!! Interrupted: Test failed, continuing from this test next run. !!!!!
========================== 2 failed in 0.07s ===========================I believe the above behavior, the combo of the flags, should work like that even if only --stepwise-skip is used.