pytest silently stops importing plugins on ImportError #1479
Description
Yesterday @craftyguy was trying to contribute something to qutebrowser, and we spent the time debugging an interesting pytest/pluggy issue instead 😆
When he tried to run pytest (tox -e py35-cov with this tox.ini), he got errors about missing arguments (added in my pytest.ini):
py35-cov runtests: commands[1] | /home/clayton/src/qutebrowser/.tox/py35-cov/bin/python -m py.test --cov --cov-report xml --cov-report=html --cov-report= tests/unit/utils/test_standarddir.py
usage: test.py [options] [file_or_dir] [file_or_dir] [...] test.py: error: unrecognized arguments: --instafail --cov --cov-report --cov-report=html --cov-report= tests/unit/utils/test_standarddir.py
inifile: /home/clayton/src/qutebrowser/pytest.ini rootdir: /home/clayton/src/qutebrowser
Looking at --version output, only a subset of the plugins was loaded:
This is pytest version 2.9.1, imported from /home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/pytest.py
setuptools registered plugins:
pytest-faulthandler-1.3.0 at /home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/pytest_faulthandler.py
pytest-catchlog-1.2.2 at /home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/pytest_catchlog.py
pytest-travis-fold-1.2.0 at /home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/pytest_travis_fold.py
However discovering them wasn't the issue:
$ ./.tox/py35-cov/bin/python -c 'import pkg_resources, pprint; pprint.pprint(list(pkg_resources.iter_entry_points("pytest11")))'
[EntryPoint.parse('pytest_faulthandler = pytest_faulthandler'),
EntryPoint.parse('pytest_catchlog = pytest_catchlog'),
EntryPoint.parse('travis-fold = pytest_travis_fold'),
EntryPoint.parse('pytest-qt = pytestqt.plugin'), EntryPoint.parse('rerunfailures = pytest_rerunfailures'),
EntryPoint.parse('pytest_mock = pytest_mock'), EntryPoint.parse('xvfb = pytest_xvfb'), EntryPoint.parse('html = pytest_html.plugin'), EntryPoint.parse('hypothesispytest = hypothesis.extra.pytestplugin'), EntryPoint.parse('pytest-bdd = pytest_bdd.plugin'),
EntryPoint.parse('pytest_cov = pytest_cov.plugin'),
EntryPoint.parse('repeat = pytest_repeat'), EntryPoint.parse('instafail = pytest_instafail')]
We then patched pytest's pluggy.py like so:
def load_setuptools_entrypoints(self, entrypoint_name):
""" Load modules from querying the specified setuptools entrypoint name.
Return the number of loaded plugins. """
from pkg_resources import iter_entry_points, DistributionNotFound
print(list(iter_entry_points(entrypoint_name)))
for ep in list(iter_entry_points(entrypoint_name)):
print(ep.name)
# is the plugin registered or blocked?
if self.get_plugin(ep.name) or self.is_blocked(ep.name):
continue
try:
plugin = ep.load()
except DistributionNotFound:
continue
except:
import traceback
traceback.print_exc()
self.register(plugin, name=ep.name)
self._plugin_distinfo.append((plugin, ep.dist))
return len(self._plugin_distinfo(printing the discovered entrypoints, printing the entrypoint name inside the for-loop, and showing all exceptions)
This finally showed us what was going wrong: pytest-qt failed to import due to some C++ ABI mismatch, but some outer layer swallowed the exception and didn't report anything at all!
./.tox/py35-cov/bin/python -m py.test --version
[EntryPoint.parse('pytest_faulthandler = pytest_faulthandler'), EntryPoint.parse('pytest_catchlog = pytest_catchlog'), EntryPoint.parse('travis-fold = pytest_travis_fold'), EntryPoint.parse('pytest-qt = pytestqt.plugin'), EntryPoint.parse('rerunfailures = pytest_rerunfailures'), EntryPoint.parse('pytest_mock = pytest_mock'), EntryPoint.parse('xvfb = pytest_xvfb'), EntryPoint.parse('html = pytest_html.plugin'), EntryPoint.parse('hypothesispytest = hypothesis.extra.pytestplugin'), EntryPoint.parse('pytest-bdd = pytest_bdd.plugin'), EntryPoint.parse('pytest_cov = pytest_cov.plugin'), EntryPoint.parse('repeat = pytest_repeat'), EntryPoint.parse('instafail = pytest_instafail')]
pytest_faulthandler
pytest_catchlog
travis-fold
pytest-qt
Traceback (most recent call last):
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 503, in load_setuptools_entrypoints
plugin = ep.load()
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2202, in load
return self.resolve()
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2208, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/pytestqt/plugin.py", line 5, in <module>
from pytestqt.logging import QtLoggingPlugin, _QtMessageCapture, Record
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/pytestqt/logging.py", line 7, in <module>
from pytestqt.qt_compat import qInstallMsgHandler, qInstallMessageHandler, \
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/pytestqt/qt_compat.py", line 80, in <module>
QtTest = _import_module('QtTest')
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/pytestqt/qt_compat.py", line 55, in _import_module
m = __import__(_root_module, globals(), locals(), [module_name], 0)
ImportError: /home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/PyQt5/QtTest.so: undefined symbol: _Z19qt_handleMouseEventP7QWindowRK7QPointFS3_6QFlagsIN2Qt11MouseButtonEES4_INS5_16KeyboardModifierEE
Traceback (most recent call last):
File "/usr/lib64/python3.5/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib64/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/py/test.py", line 4, in <module>
sys.exit(pytest.main())
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/config.py", line 39, in main
config = _prepareconfig(args, plugins)
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/config.py", line 118, in _prepareconfig
pluginmanager=pluginmanager, args=args)
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 729, in __call__
return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
_MultiCall(methods, kwargs, hook.spec_opts).execute()
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 600, in execute
return _wrapped_call(hook_impl.function(*args), self.execute)
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 249, in _wrapped_call
wrap_controller.send(call_outcome)
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/helpconfig.py", line 28, in pytest_cmdline_parse
config = outcome.get_result()
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 278, in get_result
raise ex[1].with_traceback(ex[2])
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 264, in __init__
self.result = func()
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 601, in execute
res = hook_impl.function(*args)
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/config.py", line 861, in pytest_cmdline_parse
self.parse(args)
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/config.py", line 966, in parse
self._preparse(args, addopts=addopts)
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/config.py", line 927, in _preparse
self.pluginmanager.load_setuptools_entrypoints("pytest11")
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 509, in load_setuptools_entrypoints
self.register(plugin, name=ep.name)
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/config.py", line 208, in register
ret = super(PytestPluginManager, self).register(plugin, name)
File "/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 350, in register
(plugin_name, plugin, self._name2plugin))
ValueError: Plugin already registered: pytest-qt=<module 'pytest_travis_fold' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/pytest_travis_fold.py'>
{'pastebin': <module '_pytest.pastebin' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/pastebin.py'>, 'doctest': <module '_pytest.doctest' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/doctest.py'>, '140035651859008': <_pytest.config.PytestPluginManager object at 0x7f5c9748f240>, 'monkeypatch': <module '_pytest.monkeypatch' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/monkeypatch.py'>, 'junitxml': <module '_pytest.junitxml' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/junitxml.py'>, 'pdb': <module '_pytest.pdb' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/pdb.py'>, 'assertion': <module '_pytest.assertion' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/assertion/__init__.py'>, 'helpconfig': <module '_pytest.helpconfig' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/helpconfig.py'>, 'terminal': <module '_pytest.terminal' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/terminal.py'>, 'main': <module '_pytest.main' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/main.py'>, 'nose': <module '_pytest.nose' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/nose.py'>, 'mark': <module '_pytest.mark' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/mark.py'>, 'cacheprovider': <module '_pytest.cacheprovider' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/cacheprovider.py'>, 'travis-fold': <module 'pytest_travis_fold' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/pytest_travis_fold.py'>, 'python': <module '_pytest.python' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/python.py'>, 'pytest_catchlog': <module 'pytest_catchlog' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/pytest_catchlog.py'>, 'tmpdir': <module '_pytest.tmpdir' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/tmpdir.py'>, 'genscript': <module '_pytest.genscript' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/genscript.py'>, 'pytestconfig': <_pytest.config.Config object at 0x7f5c96bc7710>, 'skipping': <module '_pytest.skipping' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/skipping.py'>, 'runner': <module '_pytest.runner' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/runner.py'>, 'resultlog': <module '_pytest.resultlog' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/resultlog.py'>, 'recwarn': <module '_pytest.recwarn' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/recwarn.py'>, 'unittest': <module '_pytest.unittest' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/unittest.py'>, 'capture': <module '_pytest.capture' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/_pytest/capture.py'>, 'pytest_faulthandler': <module 'pytest_faulthandler' from '/home/clayton/src/qutebrowser/.tox/py35-cov/lib/python3.5/site-packages/pytest_faulthandler.py'>}
Also, I noticed this part which is very confusing: ValueError: Plugin already registered: pytest-qt=<module 'pytest_travis_fold' from ...> - but maybe this was just a consequence of ignoring the exception and continuing to run the loop.