Add JSON report, confirmation stage, CI, and testpaths (v0.2.0)

- --json writes a machine-readable report (status, culprits, reproduce
  command, confirmation) for CI to gate on.
- Confirmation stage: verify the minimal set reproduces standalone and
  detect additional independent polluters instead of under-reporting.
- --no-confirm to skip the two confirmation runs.
- Report total pytest invocations across all phases.
- testpaths=[tests] so a bare pytest run ignores the self-polluting
  examples/ fixtures.
- Add CHANGELOG and a GitHub Actions test matrix (Py 3.9-3.13).

Author: python-testing-debugging

.github/workflows/ci.yml

@@ -0,0 +1,30 @@
+name: CI
+
+on:
+  push:
+    branches: [main, master]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install pytest
+        run: python -m pip install --upgrade pip pytest
+      - name: Run the test suite
+        run: PYTHONPATH=. python -m pytest -q
+      - name: Smoke-test the CLI against the bundled demo
+        run: |
+          PYTHONPATH=. python -m flake_bisect \
+            --workdir examples/polluting_demo \
+            --target 'test_target.py::test_assumes_clean_env' \
+            --json -

CHANGELOG.md

@@ -0,0 +1,30 @@
+# Changelog
+
+All notable changes to flake-bisect are documented here. This project follows
+[semantic versioning](https://semver.org/).
+
+## 0.2.0
+
+### Added
+- `--json PATH` writes a machine-readable report (`-` streams it to stdout). The
+  report carries a stable `status`, the `culprits`, a `reproduce_command`, the
+  invocation count, and the confirmation results — ready to gate CI on.
+- **Confirmation stage.** After the bisect, `flake-bisect` verifies that the
+  minimal set reproduces the failure *standalone*, then re-runs the rest of the
+  suite with the culprits removed. If the target still fails, it reports that at
+  least one more independent polluter remains instead of under-reporting.
+- `--no-confirm` skips the two confirmation runs when you want the fastest
+  possible result.
+- The stderr summary now reports the total number of pytest invocations across
+  all phases, not just bisect iterations.
+
+### Changed
+- Bare `pytest` at the repo root now collects only `tests/` (via `testpaths`),
+  so the deliberately self-polluting `examples/` fixtures no longer trip up a
+  plain test run.
+
+## 0.1.0
+
+- Initial release: delta-debugging (`ddmin`) over pytest ordering to find the
+  minimal set of tests that poison a flaky target, with a bundled order-pinning
+  plugin, sanity checks, and stable exit codes.

README.md

@@ -13,7 +13,7 @@ test ordering, so you stop guessing and start reading the right diff.
 ```
 $ python -m flake_bisect --workdir examples/polluting_demo \
                         --target test_target.py::test_assumes_clean_env
-flake-bisect 0.1.0
+flake-bisect 0.2.0
 workdir : .../examples/polluting_demo
 target  : test_target.py::test_assumes_clean_env
 Collecting tests...
@@ -30,7 +30,7 @@ Minimal poisoning set (1 test):
 Reproduce locally:
   pytest test_pollute.py::test_sets_env_flag test_target.py::test_assumes_clean_env
 
-pytest invocations during bisect: 3 (cap: 200)
+pytest invocations: 7 (bisect iterations: 3, cap: 200)
 ```
 
 A naive linear search across N candidate predecessors would take up to N runs
@@ -86,6 +86,8 @@ Or run it from inside the flake-bisect checkout with an absolute `--workdir`.
 | `--workdir`      | Run pytest from this directory (default: cwd).                       |
 | `--max-runs`     | Hard cap on pytest invocations during bisect (default: 200).         |
 | `--pytest-arg`   | Forward an arg to every pytest invocation. Repeat to pass multiple.  |
+| `--json PATH`    | Write a machine-readable report to `PATH` (`-` for stdout).          |
+| `--no-confirm`   | Skip the two post-bisect confirmation runs.                          |
 | `-v`             | Show per-iteration progress.                                         |
 
 ### Forwarding pytest options
@@ -111,9 +113,40 @@ python -m flake_bisect \
    subset is run as `pytest <subset…> <target>` in a fresh subprocess, with
    collection order pinned by a bundled internal plugin so the result doesn't
    depend on `pytest-randomly` or alphabetical ordering surprises.
-5. **Report** the minimal subset that still reproduces the failure plus a
+5. **Confirm** (unless `--no-confirm`): run the minimal set on its own to check
+   it reproduces the failure standalone, then run the *rest* of the suite with
+   the culprits removed. If the target still fails without them, there is at
+   least one more independent polluter — `flake-bisect` says so instead of
+   quietly under-reporting.
+6. **Report** the minimal subset that still reproduces the failure plus a
    copy-pasteable `pytest` command to reproduce locally.
 
+### JSON report
+
+Pass `--json report.json` (or `--json -` for stdout) to get a machine-readable
+result you can gate CI on:
+
+```json
+{
+  "tool": "flake-bisect",
+  "version": "0.2.0",
+  "status": "poisoned",
+  "exit_code": 0,
+  "target": "test_target.py::test_assumes_clean_env",
+  "culprits": ["test_pollute.py::test_sets_env_flag"],
+  "reproduce_command": "pytest test_pollute.py::test_sets_env_flag test_target.py::test_assumes_clean_env",
+  "pytest_invocations": 7,
+  "confirmation": {
+    "reproduces_standalone": true,
+    "target_clean_without_culprits": true,
+    "additional_polluters_possible": false
+  }
+}
+```
+
+The `status` field mirrors the exit codes below, so a wrapper can branch on
+either.
+
 Determinism note: `flake-bisect` cannot help with flakes caused by **time,
 threads, networking, or RNG without a fixed seed**. Those are not ordering
 bugs. If sanity check 2 doesn't reproduce the failure deterministically, the

flake_bisect/__init__.py

@@ -1,3 +1,3 @@
 """flake-bisect: find the tests that poison a flaky pytest target."""
 
-__version__ = "0.1.0"
+__version__ = "0.2.0"

flake_bisect/cli.py

@@ -3,8 +3,10 @@
 from __future__ import annotations
 
 import argparse
+import json
 import os
 import sys
+from typing import Any
 
 from flake_bisect import __version__
 from flake_bisect.bisect import ddmin
@@ -50,6 +52,18 @@ def _build_parser() -> argparse.ArgumentParser:
         metavar="ARG",
         help="Extra argument forwarded to pytest. Repeat to pass multiple.",
     )
+    p.add_argument(
+        "--json",
+        dest="json_path",
+        default=None,
+        metavar="PATH",
+        help="Write a machine-readable JSON report to PATH (use '-' for stdout).",
+    )
+    p.add_argument(
+        "--no-confirm",
+        action="store_true",
+        help="Skip the post-bisect confirmation runs (saves two pytest invocations).",
+    )
     p.add_argument(
         "-v",
         "--verbose",
@@ -59,12 +73,46 @@ def _build_parser() -> argparse.ArgumentParser:
     return p
 
 
+def _emit_json(path: str, payload: dict[str, Any]) -> None:
+    """Write the JSON report to a file, or to stdout when path == '-'."""
+    text = json.dumps(payload, indent=2, sort_keys=True)
+    if path == "-":
+        print(text)
+    else:
+        with open(path, "w", encoding="utf-8") as fh:
+            fh.write(text + "\n")
+
+
 def main(argv: list[str] | None = None) -> int:
     args = _build_parser().parse_args(argv)
 
     workdir = os.path.abspath(args.workdir)
     target = args.target
     extra = list(args.pytest_arg or [])
+    json_path = args.json_path
+
+    # Total pytest subprocess invocations across every phase (collection excluded).
+    runs = {"total": 0}
+
+    def _run(order: list[str]):
+        runs["total"] += 1
+        return run_ordered(workdir, order, extra)
+
+    def _report(status: str, exit_code: int, **fields: Any) -> int:
+        if json_path:
+            payload: dict[str, Any] = {
+                "tool": "flake-bisect",
+                "version": __version__,
+                "status": status,
+                "exit_code": exit_code,
+                "target": target,
+                "workdir": workdir,
+                "pytest_invocations": runs["total"],
+                "max_runs": args.max_runs,
+            }
+            payload.update(fields)
+            _emit_json(json_path, payload)
+        return exit_code
 
     testpaths = args.testpaths if args.testpaths else []
     print(f"flake-bisect {__version__}", file=sys.stderr)
@@ -75,59 +123,59 @@ def main(argv: list[str] | None = None) -> int:
     all_ids = collect_tests(workdir, testpaths)
     if not all_ids:
         print("ERROR: no tests collected", file=sys.stderr)
-        return 2
+        return _report("no_tests_collected", 2)
     if target not in all_ids:
         print(
             f"ERROR: target nodeid not found in collection: {target}\n"
             f"Got {len(all_ids)} nodeids; first few:\n  "
             + "\n  ".join(all_ids[:5]),
             file=sys.stderr,
         )
-        return 2
+        return _report("target_not_found", 2, collected=len(all_ids))
 
     predecessors = [n for n in all_ids if n != target]
     print(f"Collected {len(all_ids)} tests ({len(predecessors)} candidates).", file=sys.stderr)
 
     # Sanity check 1: target passes alone.
     print("Sanity check: target alone...", file=sys.stderr)
-    solo = run_ordered(workdir, [target], extra)
+    solo = _run([target])
     if target_failed(solo, target):
         print(
             f"ERROR: target fails alone — this isn't an ordering issue.\n"
             f"Outcome: {solo.outcomes.get(target)}\n"
             f"pytest exit code: {solo.exit_code}",
             file=sys.stderr,
         )
-        return 3
+        return _report("fails_alone", 3, target_outcome=solo.outcomes.get(target))
     if solo.outcomes.get(target) != "PASSED":
         print(
             f"ERROR: target was not run when invoked alone (outcome={solo.outcomes.get(target)!r}).\n"
             f"pytest exit code: {solo.exit_code}\n"
             f"--- pytest stdout ---\n{solo.stdout}",
             file=sys.stderr,
         )
-        return 3
+        return _report("target_not_run_alone", 3, target_outcome=solo.outcomes.get(target))
     print("  OK (passes alone)", file=sys.stderr)
 
     # Sanity check 2: target fails when run after everything else.
     print("Sanity check: target after full suite...", file=sys.stderr)
-    full = run_ordered(workdir, [*predecessors, target], extra)
+    full = _run([*predecessors, target])
     if not target_failed(full, target):
         print(
             f"WARNING: target did not fail in the full ordered run "
             f"(outcome={full.outcomes.get(target)!r}). Pollution may be "
             f"order- or randomness-dependent; flake-bisect cannot help here.",
             file=sys.stderr,
         )
-        return 4
+        return _report("no_pollution_reproduced", 4, target_outcome=full.outcomes.get(target))
     print(f"  OK (target outcome: {full.outcomes[target]})", file=sys.stderr)
 
     # Bisect.
     iteration = {"i": 0}
 
     def predicate(preds: list[str]) -> bool:
         iteration["i"] += 1
-        res = run_ordered(workdir, [*preds, target], extra)
+        res = _run([*preds, target])
         failed = target_failed(res, target)
         if args.verbose:
             print(
@@ -142,18 +190,66 @@ def predicate(preds: list[str]) -> bool:
         culprits = ddmin(predecessors, predicate, max_calls=args.max_runs)
     except RuntimeError as e:
         print(f"ERROR: {e}", file=sys.stderr)
-        return 5
+        return _report("max_runs_exhausted", 5, bisect_iterations=iteration["i"])
+
+    # Confirmation: (1) the minimal set reproduces the failure *standalone*, and
+    # (2) removing it from the full suite lets the target pass. If the target
+    # still fails without the culprits, there is at least one more independent
+    # polluter that this run did not surface.
+    reproduces_standalone: bool | None = None
+    target_clean_without_culprits: bool | None = None
+    if not args.no_confirm:
+        print("Confirming...", file=sys.stderr)
+        standalone = _run([*culprits, target])
+        reproduces_standalone = target_failed(standalone, target)
+        remaining = [n for n in predecessors if n not in set(culprits)]
+        without = _run([*remaining, target])
+        target_clean_without_culprits = not target_failed(without, target)
+        print(
+            f"  minimal set reproduces standalone: "
+            f"{'yes' if reproduces_standalone else 'NO'}",
+            file=sys.stderr,
+        )
+        print(
+            f"  target clean once culprits removed: "
+            f"{'yes' if target_clean_without_culprits else 'NO (more polluters remain)'}",
+            file=sys.stderr,
+        )
 
     print("")
     print(f"Minimal poisoning set ({len(culprits)} test{'s' if len(culprits) != 1 else ''}):")
     for nid in culprits:
         print(f"  {nid}")
     print("")
     print("Reproduce locally:")
-    print("  pytest " + " ".join([*culprits, target]))
+    reproduce_cmd = "pytest " + " ".join([*culprits, target])
+    print("  " + reproduce_cmd)
+    if target_clean_without_culprits is False:
+        print("")
+        print(
+            "NOTE: the target still fails with these tests removed — there is at\n"
+            "      least one more independent polluter. Re-run flake-bisect after\n"
+            "      fixing this set to find the next one."
+        )
     print("")
-    print(f"pytest invocations during bisect: {iteration['i']} (cap: {args.max_runs})", file=sys.stderr)
-    return 0
+    print(
+        f"pytest invocations: {runs['total']} "
+        f"(bisect iterations: {iteration['i']}, cap: {args.max_runs})",
+        file=sys.stderr,
+    )
+
+    return _report(
+        "poisoned",
+        0,
+        culprits=culprits,
+        reproduce_command=reproduce_cmd,
+        bisect_iterations=iteration["i"],
+        confirmation={
+            "reproduces_standalone": reproduces_standalone,
+            "target_clean_without_culprits": target_clean_without_culprits,
+            "additional_polluters_possible": target_clean_without_culprits is False,
+        },
+    )
 
 
 if __name__ == "__main__":

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "flake-bisect"
-version = "0.1.0"
+version = "0.2.0"
 description = "Bisect a pytest suite to find the test(s) poisoning a flaky target."
 readme = "README.md"
 license = { file = "LICENSE" }
@@ -31,3 +31,9 @@ Source = "https://github.com/python-testing-debugging/flake-bisect"
 [tool.setuptools.packages.find]
 include = ["flake_bisect*"]
 exclude = ["tests*", "examples*"]
+
+[tool.pytest.ini_options]
+# Only collect flake-bisect's own tests. The examples/ suites are deliberately
+# self-polluting fixtures — they are meant to be run *by* flake-bisect, not by
+# pytest at the repo root.
+testpaths = ["tests"]