Skip to content

--last-failed can silently skip failing custom items with brackets in their names #15045

Description

@host452b

When a custom collector creates an Item whose name contains brackets, --last-failed can omit that item even though it failed in the previous run. If another previously failing item has been fixed, pytest reports success while the bracket-named item still fails in a full run.

Reproduced on main at 99ab2ac (9.2.0.dev335+g99ab2accc), Linux x86_64, Python 3.12.3, pluggy 1.6.0. Plugin autoload is disabled. The reproducer only needs pytest and the standard library. The documented YAML collector exhibits the same behavior; pytest 9.1.1 handles that YAML scenario correctly.

Save this as reproduce.py and run it using the Python environment containing the current development version of pytest:

import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile

conftest = '''import json
import pytest


def pytest_collect_file(parent, file_path):
    if file_path.name == "test_cases.json":
        return Cases.from_parent(parent, path=file_path)


class Cases(pytest.File):
    def collect(self):
        for name, passed in json.loads(self.path.read_text()).items():
            yield Case.from_parent(self, name=name, passed=passed)


class Case(pytest.Item):
    def __init__(self, *, passed, **kwargs):
        super().__init__(**kwargs)
        self.passed = passed

    def runtest(self):
        assert self.passed
'''

with tempfile.TemporaryDirectory() as directory:
    root = Path(directory)
    (root / 'conftest.py').write_text(conftest)
    (root / 'pytest.ini').write_text('[pytest]\n')
    cases = root / 'test_cases.json'
    env = dict(os.environ, PYTEST_DISABLE_PLUGIN_AUTOLOAD='1')
    results = []
    for label, data, args in [
        ('initial', {'a_bad[one]': False, 'b_fixed': False}, []),
        ('last-failed after fixing b_fixed', {'a_bad[one]': False, 'b_fixed': True}, ['--lf']),
        ('full run with same input', {'a_bad[one]': False, 'b_fixed': True}, []),
    ]:
        cases.write_text(json.dumps(data))
        run = subprocess.run([sys.executable, '-m', 'pytest', '-v', '--tb=no', *args], cwd=root, env=env)
        print(f'{label}: exit {run.returncode}', flush=True)
        results.append(run.returncode)
    print(f'Exit sequence: {results}', flush=True)

Expected exit sequence: [1, 1, 1] (initial failures, rerun after fixing only b_fixed, full run).
Actual exit sequence: [1, 0, 1]. The second invocation collects only one item:

collecting ... collected 1 item
run-last-failure: rerun previous 1 failure
test_cases.json::b_fixed PASSED
1 passed

The subsequent full run reports a_bad[one] FAILED and b_fixed PASSED.

The failure appears to be in NodeId identity across the string cache boundary. The live custom item has names=("a_bad[one]",), params=None; parsing its cached string gives names=("a_bad",), params="one". These have identical public nodeid strings but compare and hash differently. LFPluginCollWrapper filters items using these structured IDs. If there is only the bracket-named failure, the no-matching-failure fallback runs everything; the second, matching failure is what exposes the omission and false success.

I plan to follow up with a focused fix and regression tests. This report and reproducer were prepared with OpenAI Codex assistance; the exit sequence above was verified by running the script.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions