Is there a way to run doctests and continue on failure? #3149
Description
Say you have a file named t.py:
"""
>>> print(1)
>>> print(1)
>>> print(1)
"""
if __name__ == "__main__":
import doctest
doctest.testmod()When I run this, I would get:
$ python t.py
**********************************************************************
File "t.py", line 2, in __main__
Failed example:
print(1)
Expected nothing
Got:
1
**********************************************************************
File "t.py", line 3, in __main__
Failed example:
print(1)
Expected nothing
Got:
1
**********************************************************************
File "t.py", line 4, in __main__
Failed example:
print(1)
Expected nothing
Got:
1
**********************************************************************
1 items had failures:
3 of 3 in __main__
***Test Failed*** 3 failures.
However, when I run with pytest, I would only get the first failure:
$ pytest t.py
========================================================= test session starts =========================================================
platform linux2 -- Python 2.7.14, pytest-3.3.2, py-1.5.2, pluggy-0.5.2
rootdir: /auto/cnvtvws/wlee/fcnvt/dates, inifile: setup.cfg
plugins: xdist-1.20.1, forked-0.2, flakes-1.0.1, flake8-0.9, cov-2.5.1, hypothesis-3.38.5, pep8-1.0.6
collected 2 items
t.py sF [100%]
======================================================= short test summary info =======================================================
FAIL t.py::t
============================================================== FAILURES ===============================================================
_____________________________________________________________ [doctest] t _____________________________________________________________
001
002 >>> print(1)
Expected nothing
Got:
1
.../t.py:2: DocTestFailure
================================================= 1 failed, 1 skipped in 0.01 seconds =================================================
I've tried to dig through all the options form the documentation but I wan't able to figure it out. Is there a way to have pytest report all the failures at once? Otherwise I'd have to fix the failures one by one, which gets tedious. I tried to put in various entries in --doctest-report='none' but can't get it to work.
Here is the relevant section in setup.cfg:
[tool:pytest]
addopts = -rf --strict --doctest-modules --flake8 --doctest-glob='*.rst'
doctest_optionflags = NORMALIZE_WHITESPACE
maxfail = 5
Here is my pip list:
alabaster (0.7.10)
asn1crypto (0.24.0)
attrs (17.3.0)
Babel (2.5.3)
certifi (2017.11.5)
cffi (1.11.4)
chardet (3.0.4)
configparser (3.5.0)
coverage (4.4.2)
cryptography (2.1.4)
docutils (0.14)
enum34 (1.1.6)
execnet (1.2.0)
flake8 (3.5.0)
funcsigs (1.0.2)
hypothesis (3.38.5)
idna (2.6)
imagesize (0.7.1)
ipaddress (1.0.19)
Jinja2 (2.10)
llvmlite (0.15.0)
MarkupSafe (1.0)
mccabe (0.6.1)
numba (0.30.1+0.g8c1033f.dirty)
numpy (1.11.3)
pep8 (1.7.1)
pip (9.0.1)
pluggy (0.6.0)
py (1.5.2)
pycodestyle (2.3.1)
pycparser (2.18)
pyflakes (1.6.0)
Pygments (2.2.0)
Pympler (0.5)
pyOpenSSL (17.5.0)
PySocks (1.6.7)
pytest (3.3.2)
pytest-cache (1.0)
pytest-cov (2.5.1)
pytest-flake8 (0.9)
pytest-flakes (1.0.1)
pytest-forked (0.2)
pytest-pep8 (1.0.6)
pytest-runner (3.0)
pytest-xdist (1.20.1)
python-dateutil (2.4.1)
pytz (2015.7)
requests (2.18.4)
setuptools (38.4.0)
singledispatch (3.4.0.3)
six (1.11.0)
snowballstemmer (1.2.1)
Sphinx (1.6.6)
sphinxcontrib-websupport (1.0.1)
typing (3.6.2)
urllib3 (1.22)
wheel (0.30.0)
zope.interface (4.4.3)