-p plugin not working, fails to early-load plugin #4718
Description
Operating systems: Ubuntu 16.04, macOS Sierra, macOS High Sierra
Python: 3.7.1
Pytest:
$ pytest --version
This is pytest version 4.2.0, imported from /usr/local/lib/python3.7/site-packages/pytest.py
setuptools registered plugins:
pytest-django-3.4.7 at /usr/local/lib/python3.7/site-packages/pytest_django/plugin.py
pytest-cov-2.6.1 at /usr/local/lib/python3.7/site-packages/pytest_cov/plugin.py
pysoa-0.56.0 at /usr/local/lib/python3.7/site-packages/pysoa/test/plugins/pytest/fixtures.py
pysoa-0.56.0 at /usr/local/lib/python3.7/site-packages/pysoa/test/plugins/pytest/plans.py
pytest --help describes an option for early-loading plugins:
-p name early-load given plugin (multi-allowed). To avoid
loading of plugins, use the `no:` prefix, e.g.
`no:doctest`.
I hoped to use this to early-load the code coverage plugin pytest-cov. (The motivation is that pytest-cov should be loaded before any other plugins, but on some of our machines pytest-django is loading first, resulting in about a 15% coverage difference, which causes tests to fail on those machines because we have --fail-under set for enforcement of code coverage.) However, as discussed in #935, -p is broken and no longer works. Here are all the different approaches I tried:
# disables plugin
$ pytest -p pytest_cov
usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov-report --cov-branch --cov-fail-under=90 --cov=.......
# does not recognize it
$ pytest -p pytest-cov
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 530, in import_plugin
__import__(importspec)
ModuleNotFoundError: No module named 'pytest-cov'
# does not recognize it
$ pytest -p cov
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 530, in import_plugin
__import__(importspec)
ModuleNotFoundError: No module named 'cov'
# finally no error, but does not load it early
$ pytest -p coverage
================= test session starts =================
platform linux -- Python 3.7.1, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
Django settings: xxxx.settings.dev_local (from environment variable)
rootdir: /srv/xxxx, inifile: setup.cfg
plugins: django-3.4.6, cov-2.6.1, pysoa-0.56.0
collected 256 items
...
xxxx/settings/__init__.py 36 36 0 0 0.0% 1-135
xxxx/settings/dev_local.py 33 33 6 0 0.0% 1-125
xxxx/settings/jenkins_test.py 24 24 4 0 0.0% 1-131
xxxx/settings/production_docker.py 23 23 9 0 0.0% 1-125
...
# gives an error about its already being loaded
$ pytest -p pytest_cov.plugin
Traceback (most recent call last):
File "/usr/local/bin/pytest", line 11, in <module>
sys.exit(main())
...
ValueError: Plugin already registered: pytest_cov=<module 'pytest_cov.plugin' from '/usr/local/lib/python3.7/site-packages/pytest_cov/plugin.py'>
{'140434823318496': <_pytest.config.PytestPluginManager object at 0x7fb987c20be0>, 'pytestconfig': <_pytest.config.Config object at 0x7fb98498c5c0> ......
I'm not sure if this is a regression or if this command-line argument has always been mis-documented and wasn't actually intended for early-loading plugins. Either way, it's a bug. The -p argument conflicts with the entrypoints plugin loading system and there is no way to early-load a plugin unless it doesn't have an entrypoint.