Initial commit: flake-bisect

python-testing-debugging · python-testing-debugging · commit 7aa567686987 · 2026-05-26T23:27:04.000+02:00
Delta-debugging CLI that finds the minimal set of preceding pytest tests
that poison a flaky target. Wraps Zeller's ddmin around subprocess pytest
runs, pinning collection order via a bundled internal plugin so results
are stable regardless of pytest-randomly or alphabetical ordering.

- flake_bisect/ core (bisect, runner, order plugin, CLI)
- examples/polluting_demo/ minimal reproducer suite
- tests/ unit + end-to-end coverage
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,18 @@
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+.pytest_cache/
+.venv/
+venv/
+.env
+.coverage
+htmlcov/
+build/
+dist/
+.DS_Store
+.idea/
+.vscode/
+AGENTS.md
+.claude/
+CLAUDE.md
diff --git a/LICENSE b/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 python-testing-debugging
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/README.md b/README.md
@@ -0,0 +1,155 @@
+# flake-bisect
+
+Find the pytest test(s) that **poison** a flaky target.
+
+You have a test that passes when you run it alone but fails as part of the full
+suite. Some other test mutates global state — `os.environ`, a singleton, a
+module-level cache, a database row, the current working directory, a registered
+signal handler — and your target is the one that notices. `flake-bisect`
+narrows the polluter down to a minimal set using
+[delta-debugging](https://www.st.cs.uni-saarland.de/papers/tse2002/) over the
+test ordering, so you stop guessing and start reading the right diff.
+
+```
+$ python -m flake_bisect --workdir examples/polluting_demo \
+                        --target test_target.py::test_assumes_clean_env
+flake-bisect 0.1.0
+workdir : .../examples/polluting_demo
+target  : test_target.py::test_assumes_clean_env
+Collecting tests...
+Collected 8 tests (7 candidates).
+Sanity check: target alone...
+  OK (passes alone)
+Sanity check: target after full suite...
+  OK (target outcome: FAILED)
+Bisecting 7 candidate predecessors...
+
+Minimal poisoning set (1 test):
+  test_pollute.py::test_sets_env_flag
+
+Reproduce locally:
+  pytest test_pollute.py::test_sets_env_flag test_target.py::test_assumes_clean_env
+
+pytest invocations during bisect: 3 (cap: 200)
+```
+
+A naive linear search across N candidate predecessors would take up to N runs
+of the suite. `flake-bisect` typically converges in **O(log N)** pytest
+invocations when there is a single polluter, and stays sub-linear with a small
+number of polluters.
+
+## Why this exists
+
+The Python testing & debugging community has converged on a clear playbook for
+non-flaky suites: ban global state in tests, use fixtures with `monkeypatch`,
+isolate the DB per test, run with [`pytest-randomly`][pytest-randomly] in CI to
+surface ordering bugs early. The hard part is what to do when CI catches one.
+The failing line tells you *what* broke; it never tells you *who* set the
+landmine 200 tests earlier.
+
+`flake-bisect` does that last mile: given a known-flaky target, it points at
+the test that poisons it.
+
+[pytest-randomly]: https://github.com/pytest-dev/pytest-randomly
+
+## Running it
+
+`flake-bisect` is a self-contained Python package with no third-party
+dependencies. It needs `pytest` available in the same Python environment as
+the project you're bisecting (it shells out to `python -m pytest`).
+
+Clone the repo and run from source:
+
+```bash
+git clone https://github.com/python-testing-debugging/flake-bisect.git
+cd flake-bisect
+python -m flake_bisect --help
+```
+
+To use it against your own project, point `--workdir` at your project root and
+add `flake-bisect` to `PYTHONPATH` so the module is importable:
+
+```bash
+PYTHONPATH=/path/to/flake-bisect python -m flake_bisect \
+    --workdir /path/to/your/project \
+    --target tests/test_widgets.py::test_render_safely
+```
+
+Or run it from inside the flake-bisect checkout with an absolute `--workdir`.
+
+### Common flags
+
+| Flag             | Purpose                                                              |
+| ---------------- | -------------------------------------------------------------------- |
+| `--target`       | Required. The flaky test's nodeid as pytest reports it.              |
+| `--testpaths`    | Limit candidate predecessors to specific paths (otherwise full suite).|
+| `--workdir`      | Run pytest from this directory (default: cwd).                       |
+| `--max-runs`     | Hard cap on pytest invocations during bisect (default: 200).         |
+| `--pytest-arg`   | Forward an arg to every pytest invocation. Repeat to pass multiple.  |
+| `-v`             | Show per-iteration progress.                                         |
+
+### Forwarding pytest options
+
+If your project needs particular pytest options to even collect (a `-p` plugin,
+`-o` override, marker filter, etc.), forward them with repeated `--pytest-arg`:
+
+```bash
+python -m flake_bisect \
+    --target tests/test_x.py::test_y \
+    --pytest-arg -o --pytest-arg "addopts=" \
+    --pytest-arg -m --pytest-arg "not slow"
+```
+
+## How it works
+
+1. **Collect** all nodeids in the suite via `pytest --collect-only`.
+2. **Sanity check 1**: run the target alone; bail out if it fails (then the
+   issue isn't ordering, it's the test itself).
+3. **Sanity check 2**: run `[…all other tests…, target]` in order; bail out if
+   the target passes (no reproducible pollution to bisect).
+4. **Delta-debug** the predecessor list with Zeller's `ddmin`. Each candidate
+   subset is run as `pytest <subset…> <target>` in a fresh subprocess, with
+   collection order pinned by a bundled internal plugin so the result doesn't
+   depend on `pytest-randomly` or alphabetical ordering surprises.
+5. **Report** the minimal subset that still reproduces the failure plus a
+   copy-pasteable `pytest` command to reproduce locally.
+
+Determinism note: `flake-bisect` cannot help with flakes caused by **time,
+threads, networking, or RNG without a fixed seed**. Those are not ordering
+bugs. If sanity check 2 doesn't reproduce the failure deterministically, the
+bug is somewhere else and the CLI will say so.
+
+## Exit codes
+
+| Code | Meaning                                                          |
+| ---: | ---------------------------------------------------------------- |
+|    0 | Bisect completed; poisoning set printed.                         |
+|    2 | Collection problem (no tests, target nodeid not found, ...).    |
+|    3 | Target fails when run alone — not an ordering issue.            |
+|    4 | Target passes in the full ordered run — no pollution reproduced. |
+|    5 | `--max-runs` budget exhausted.                                   |
+
+These are stable; CI can branch on them.
+
+## Demo
+
+The `examples/polluting_demo/` directory contains a six-test suite with one
+polluter and one target. Use it to verify the tool runs in your environment:
+
+```bash
+python -m flake_bisect \
+    --workdir examples/polluting_demo \
+    --target test_target.py::test_assumes_clean_env
+```
+
+You should see `test_pollute.py::test_sets_env_flag` named as the culprit.
+
+## Background reading
+
+Deeper material on flaky tests, pytest internals, isolation patterns, and the
+delta-debugging algorithm lives at
+[python-testing-debugging.com](https://python-testing-debugging.com).
+
+## License
+
+[MIT](LICENSE)
diff --git a/examples/polluting_demo/README.md b/examples/polluting_demo/README.md
@@ -0,0 +1,17 @@
+# polluting_demo
+
+A miniature pytest suite that reproduces a classic test-pollution flake:
+
+- `test_pollute.py::test_sets_env_flag` writes to `os.environ`.
+- `test_target.py::test_assumes_clean_env` asserts the same env var is **not** set.
+
+The target passes when run alone but fails after the polluter has run. Use this
+suite as a fixture to exercise `flake-bisect`:
+
+```bash
+python -m flake_bisect \
+    --workdir examples/polluting_demo \
+    --target test_target.py::test_assumes_clean_env
+```
+
+Expected: flake-bisect narrows down to `test_pollute.py::test_sets_env_flag`.
diff --git a/examples/polluting_demo/conftest.py b/examples/polluting_demo/conftest.py
@@ -0,0 +1 @@
+"""Empty conftest so pytest treats this directory as a rootdir on its own."""
diff --git a/examples/polluting_demo/test_innocent_a.py b/examples/polluting_demo/test_innocent_a.py
@@ -0,0 +1,6 @@
+def test_addition():
+    assert 1 + 1 == 2
+
+
+def test_string_format():
+    assert f"{'flake':>10}" == "     flake"
diff --git a/examples/polluting_demo/test_innocent_b.py b/examples/polluting_demo/test_innocent_b.py
@@ -0,0 +1,10 @@
+def test_list_extend():
+    xs = [1, 2]
+    xs.extend([3, 4])
+    assert xs == [1, 2, 3, 4]
+
+
+def test_dict_merge():
+    a = {"x": 1}
+    b = {"y": 2}
+    assert {**a, **b} == {"x": 1, "y": 2}
diff --git a/examples/polluting_demo/test_innocent_c.py b/examples/polluting_demo/test_innocent_c.py
@@ -0,0 +1,7 @@
+def test_set_intersection():
+    assert {1, 2, 3} & {2, 3, 4} == {2, 3}
+
+
+def test_tuple_unpack():
+    a, b = (10, 20)
+    assert a + b == 30
diff --git a/examples/polluting_demo/test_pollute.py b/examples/polluting_demo/test_pollute.py
@@ -0,0 +1,9 @@
+import os
+
+POLLUTION_KEY = "FLAKE_BISECT_DEMO_POLLUTED"
+
+
+def test_sets_env_flag():
+    """Innocent-looking test that forgets to clean up after itself."""
+    os.environ[POLLUTION_KEY] = "1"
+    assert os.environ[POLLUTION_KEY] == "1"
diff --git a/examples/polluting_demo/test_target.py b/examples/polluting_demo/test_target.py
@@ -0,0 +1,10 @@
+import os
+
+POLLUTION_KEY = "FLAKE_BISECT_DEMO_POLLUTED"
+
+
+def test_assumes_clean_env():
+    """Fails iff a previous test set POLLUTION_KEY in os.environ."""
+    assert POLLUTION_KEY not in os.environ, (
+        f"another test polluted os.environ with {POLLUTION_KEY}"
+    )
diff --git a/flake_bisect/__init__.py b/flake_bisect/__init__.py
@@ -0,0 +1,3 @@
+"""flake-bisect: find the tests that poison a flaky pytest target."""
+
+__version__ = "0.1.0"
diff --git a/flake_bisect/__main__.py b/flake_bisect/__main__.py
@@ -0,0 +1,3 @@
+from flake_bisect.cli import main
+
+raise SystemExit(main())
diff --git a/flake_bisect/_order_plugin.py b/flake_bisect/_order_plugin.py
@@ -0,0 +1,22 @@
+"""Pytest plugin: enforce a deterministic test order from FLAKE_BISECT_ORDER.
+
+The env var holds a newline-separated list of nodeids. Items not in the list are
+dropped from the run; items in the list execute in listed order. This makes the
+subprocess deterministic regardless of plugins like pytest-randomly that the
+target project may have installed.
+"""
+
+from __future__ import annotations
+
+import os
+
+
+def pytest_collection_modifyitems(config, items):  # noqa: ARG001
+    raw = os.environ.get("FLAKE_BISECT_ORDER", "")
+    if not raw:
+        return
+    desired = [line for line in raw.split("\n") if line]
+    index = {nid: i for i, nid in enumerate(desired)}
+    kept = [it for it in items if it.nodeid in index]
+    kept.sort(key=lambda it: index[it.nodeid])
+    items[:] = kept
diff --git a/flake_bisect/bisect.py b/flake_bisect/bisect.py
@@ -0,0 +1,77 @@
+"""Delta-debugging (ddmin) over an ordered list of pytest nodeids."""
+
+from __future__ import annotations
+
+from collections.abc import Callable, Sequence
+from typing import TypeVar
+
+T = TypeVar("T")
+
+
+def ddmin(
+    items: Sequence[T],
+    predicate: Callable[[list[T]], bool],
+    *,
+    max_calls: int | None = None,
+) -> list[T]:
+    """Zeller-style delta-debugging.
+
+    Returns a minimal sublist of `items` (preserving order) for which `predicate`
+    returns True. Assumes `predicate(list(items))` already returns True.
+
+    Raises `RuntimeError` if `max_calls` is exceeded.
+    """
+    current: list[T] = list(items)
+    if not current:
+        return current
+
+    calls = 0
+    cache: dict[tuple, bool] = {}
+
+    def test(subset: list[T]) -> bool:
+        nonlocal calls
+        key = tuple(subset)
+        if key in cache:
+            return cache[key]
+        if max_calls is not None and calls >= max_calls:
+            raise RuntimeError(
+                f"flake-bisect: exceeded max-runs={max_calls} during ddmin"
+            )
+        calls += 1
+        result = predicate(subset)
+        cache[key] = result
+        return result
+
+    n = 2
+    while len(current) >= 2:
+        chunk_size = max(len(current) // n, 1)
+        chunks = [current[i : i + chunk_size] for i in range(0, len(current), chunk_size)]
+
+        # 1) Try each subset alone.
+        narrowed = False
+        for chunk in chunks:
+            if chunk and test(chunk):
+                current = chunk
+                n = 2
+                narrowed = True
+                break
+        if narrowed:
+            continue
+
+        # 2) Try each complement.
+        for chunk in chunks:
+            chunk_set = set(chunk)
+            complement = [x for x in current if x not in chunk_set]
+            if complement and test(complement):
+                current = complement
+                n = max(n - 1, 2)
+                narrowed = True
+                break
+        if narrowed:
+            continue
+
+        if n >= len(current):
+            break
+        n = min(n * 2, len(current))
+
+    return current
diff --git a/flake_bisect/cli.py b/flake_bisect/cli.py
diff --git a/flake_bisect/runner.py b/flake_bisect/runner.py
diff --git a/pyproject.toml b/pyproject.toml
diff --git a/tests/__init__.py b/tests/__init__.py
diff --git a/tests/test_bisect.py b/tests/test_bisect.py
diff --git a/tests/test_end_to_end.py b/tests/test_end_to_end.py

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1 @@`
	`1`	`+"""Empty conftest so pytest treats this directory as a rootdir on its own."""`