This was originally noted in #9892, but there is a more targeted MRE here.
MRE
Running pytest --doctest-modules test_x.py with x=1, 2 or 3 results in the following pattern of failing and passing tests. The test inside test_wo_number incorrectly passes only on the cases (test_1.py and test_2.py) where a failing test that uses NUMBER (test_w_off_by_2_number) also exists in the same module. This leakage does not occur when there are tests that pass that utilize NUMBER (test_3.py)
# test_1.py
def test_w_number():
"""
Correctly Passes
Examples
--------
>>> 1. # doctest: +NUMBER
0.
"""
def test_w_off_by_2_number():
"""
Correctly Fails
Examples
--------
>>> 0. # doctest: +NUMBER
2.
"""
def test_wo_number():
"""
Incorrectly Passes
Examples
--------
>>> 1.
0.
"""# test_2.py
def test_w_off_by_2_number():
"""
Correctly Fails
Examples
--------
>>> 0. # doctest: +NUMBER
2.
"""
def test_wo_number():
"""
Incorrectly Passes
Examples
--------
>>> 1.
0.
"""# test_3.py
def test_w_number():
"""
Correctly Passes
Examples
--------
>>> 1. # doctest: +NUMBER
0.
"""
def test_wo_number():
"""
Correctly Fails
Examples
--------
>>> 1.
0.
"""pip list
Package Version
---------- -------
attrs 21.4.0
iniconfig 1.1.1
numpy 1.22.3
packaging 21.3
pip 22.0.4
pluggy 1.0.0
py 1.11.0
pyparsing 3.0.8
pytest 7.1.2
setuptools 41.2.0
tomli 2.0.1
system version
CentOS 7.9
pip listfrom the virtual environment you are usingThis was originally noted in #9892, but there is a more targeted MRE here.
MRE
Running
pytest --doctest-modules test_x.pywithx=1, 2 or 3results in the following pattern of failing and passing tests. The test insidetest_wo_numberincorrectly passes only on the cases (test_1.py and test_2.py) where a failing test that usesNUMBER(test_w_off_by_2_number) also exists in the same module. This leakage does not occur when there are tests that pass that utilizeNUMBER(test_3.py)pip list
system version
CentOS 7.9