Skip to content

Unrelated testcases fail when total number of tests is increased beyond 367 #4275

@sumezulike

Description

@sumezulike

Describe the bug

An extremely strange bug concerning development:
If the number of testcases collected by pytest exceeds 367, and pytest is run with the argument --numprocesses with a value of 4, 6, or 8, the test cases test_get_sources_with_stdin_filename_and_exclude and test_get_sources_with_stdin_filename_and_extend_exclude fail.

To Reproduce

  • Clone this repo, as of writing the most recent commit on main is 1abcffc.
  • Follow the guide to install the dependencies.
  • Run tox -e py -- --numprocesses 4 and verify that all tests pass.
  • Note the number of collected test items, currently it is 355.
  • Raise the number to 368, for example by creating files in tests/data/cases, by adding parameters to a parameterized test or by appending the below to a test file, e.g tests/test_black.py
def test_nothing_1():
    pass
def test_nothing_2():
    pass
def test_nothing_3():
    pass
def test_nothing_4():
    pass
def test_nothing_5():
    pass
def test_nothing_6():
    pass
def test_nothing_7():
    pass
def test_nothing_8():
    pass
def test_nothing_9():
    pass
def test_nothing_10():
    pass
def test_nothing_11():
    pass
def test_nothing_12():
    pass
def test_nothing_13():
    pass
  • Run tox -e py -- --numprocesses 4 and expect the following failures:
FAILED tests/test_black.py::TestFileCollection::test_get_sources_with_stdin_filename_and_exclude - AssertionError: assert [] == [PosixPath('_...xclude/a.py')]
FAILED tests/test_black.py::TestFileCollection::test_get_sources_with_stdin_filename_and_extend_exclude - AssertionError: assert [] == [PosixPath('_...xclude/a.py')]
  • Remove one of the test cases.
  • Run tox -e py -- --numprocesses 4 and verify that all tests pass again.

Expected behavior

Well, not that.

Environment

$ black --version
black, 24.2.1.dev11+g1abcffc.d20240314 (compiled: no)
Python (CPython) 3.11.8

Pytest environment output:

platform linux -- Python 3.11.8, pytest-8.1.1, pluggy-1.4.0
cachedir: .tox/py/.pytest_cache
rootdir: /home/samson/PycharmProjects/black
configfile: pyproject.toml
plugins: xdist-3.5.0, cov-4.1.0

OS: EndeavourOS Linux x86_64
Kernel: 6.7.9-arch1-1

Additional context

I've reproduced this on one other machine.
That's not a large sample size, but the consistency of exactly those two tests failing for the same --numprocesses values and the same number of tests makes me think it's a real issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    T: bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions