incorrect case in test exclusion via marks causes collection crash #4583
Description
Im going to preface this by saying I stumbled across this and it can be arguably caused by user error. That said, I am reporting this due to the fashion in which pytest fails.
Issue
When deselecting tests by test mark, if NOT is used rather than not, pytest crashes with a lengthy stacktrace.
Action root@5de4510ee14c:/potato# pytest -m "NOT database"
expected behavior In this case, i expected pytest to handle as if i used not, deselecting my database tests and running the rest of the suite
- I would also accept an argument error of some form, a graceful error would suffice.
actual behavior
pytest -m "NOT (database)"
================================================= test session starts ==================================================
platform linux -- Python 3.6.6, pytest-4.0.2, py-1.7.0, pluggy-0.8.0
rootdir: /potato, inifile: pytest.ini
collecting ... INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/_pytest/main.py", line 185, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/_pytest/main.py", line 224, in _main
INTERNALERROR> config.hook.pytest_collection(session=session)
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/pluggy/hooks.py", line 284, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 67, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 61, in <lambda>
INTERNALERROR> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/_pytest/main.py", line 234, in pytest_collection
INTERNALERROR> return session.perform_collect()
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/_pytest/main.py", line 450, in perform_collect
INTERNALERROR> session=self, config=self.config, items=items
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/pluggy/hooks.py", line 284, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 67, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 61, in <lambda>
INTERNALERROR> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/_pytest/mark/__init__.py", line 153, in pytest_collection_modifyitems
INTERNALERROR> deselect_by_mark(items, config)
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/_pytest/mark/__init__.py", line 141, in deselect_by_mark
INTERNALERROR> if matchmark(item, matchexpr):
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/_pytest/mark/legacy.py", line 71, in matchmark
INTERNALERROR> return eval(markexpr, {}, MarkMapping.from_item(colitem))
INTERNALERROR> File "<string>", line 1, in <module>
INTERNALERROR> TypeError: 'bool' object is not callable
============================================= no tests ran in 0.02 seconds =============================================
- Include a detailed description of the bug or suggestion
pip list
Package Version
-------------- -------
atomicwrites 1.2.1
attrs 18.2.0
more-itertools 5.0.0
pip 18.0
pluggy 0.8.0
py 1.7.0
pytest 4.0.2
setuptools 40.0.0
six 1.12.0
wheel 0.31.1
-
pip listof the virtual environment you are using - pytest and operating system versions
pytest version4.0.2,
OS: Windows 10 x64, Linux (Dockerpython:3.6container)
Python versions: 3.7.1 x64, 3.6.6 - Minimal example if possible
Below is a minimal setup that results in this crash
Example pytest.ini
[pytest]
markers =
database: marks a test as a database testexample test file test_potato.py
from pytest import mark
@mark.database
def test_foo():
assert 2+2 == 4
def test_bar():
assert 4*4 == 16example falsifying pytest invocation:
pytest -m "NOT (database)"-- its worth noting that pytest -m "not (database)" works as expected, and does not result in the above crash