[antonwolfy]

Summary

Follow-up to the review of #238, where repo run --batch X on a batch whose finished cell holds a truncated result.json dies with a bare JSONDecodeError instead of reporting that cell (review, item 2 of B3). #238 narrowed its changelog; this changes the behaviour.

On main, run_cell reads a finished cell's result.json with a bare json.loads, and run_batch's per-cell handler does not catch ValueError, so the error ends the whole command. Rerunning a two-rep batch (--arm guidance:raw --task fix_add --reps 2) after writing a bad result.json into the r1 cell:

r1 result.json	main	this branch
{not json	✗ Expecting property name enclosed in double quotes: line 1 column 2 (char 1), no summary line, exit 1	✗ fix_add__guidance-raw__m-1__r1: runs/b1/fix_add__guidance-raw__m-1__r1/result.json: not valid JSON (Expecting property name …); remove it to run the cell again, or rerun with --force, which replaces every cell of this run's plan, then 0 ran, 1 already done, 1 crashed, …, exit 1
\xff{} (not UTF-8)	✗ 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte, exit 1	✗ fix_add__guidance-raw__m-1__r1: …/result.json: not valid UTF-8 text; remove it …, the other cell reused, exit 1
[]	refused before any cell runs: ✗ runs/b1 holds cells of another execution (unrecorded) than this run (…); use another --batch, or --force to replace them, exit 1	✗ fix_add__guidance-raw__m-1__r1: …/result.json: not a JSON object; remove it …, the other cell reused, exit 1

A finished cell of a task this host cannot run (its host requirements unmet) takes another path. run_batch keeps that cell, even with --force, and on main counts it as already done because its result.json exists, without reading it. Here both fix_race cells finished on a capable host, then the batch was rerun with --task fix_add --task fix_race --reps 2 on a 2-CPU AMD host:

fix_race cells	main	this branch
r2 cut in half	0 ran, 4 already done, 0 crashed, …, exit 0 (with --force: 2 ran, 2 already done, …, exit 0)	✗ fix_race__guidance-raw__m-1__r2: …/result.json: not valid JSON (Unterminated string …); this host cannot run the task, so --force keeps the file; …, then 0 ran, 3 already done, 1 crashed, …, exit 1
r1 [], r2 cut in half, --force	✗ runs/b1 holds cells of another execution (unrecorded) than this run (…), exit 1	✗ fix_race__guidance-raw__m-1__r1: …/result.json: not a JSON object; this host cannot run the task, so --force keeps the file; remove it, and the cell runs on a host that meets the task's requirements, ✗ fix_race__guidance-raw__m-1__r2: …/result.json: not valid JSON (Unterminated string …); …, then 2 ran, 0 already done, 2 crashed, …, exit 1

With this PR's first commit alone, check_batch_executions skipped the [] file as well, so the second row printed 2 ran, 2 already done, … and exited 0 (review comment); the second commit reports both cells.

Changes

• dokimi/dokimi/repos/runner.py: run_cell reads a finished cell through a new _finished_result, which raises UnreadableResult (a PrepError, which run_batch already reports per cell) when the file is not UTF-8, not JSON or not a JSON object. The message starts with the file's path and names both ways back: remove the file, or --force. The cell is logged as ✗ <cell>: <path>: ..., counted among the crashed cells, and the command exits 1 as for any other crashed cell; the other cells still run or are reused. The file is neither changed nor removed, and the cell is not rerun on its own.
• run_batch's branch for a task this host cannot run moves into _unsupported_cell, which reads a kept result.json through the same helper. When that file cannot be read, run_batch records the cell's error and logs it the same way, with or without --force. The message says --force keeps the file on this host and that removing it lets the cell run on a host that meets the task's requirements. The file stays as it is, and the cell is neither rerun nor marked unsupported. A readable kept result still counts as already done.
• check_batch_executions reads through the same helper: as before, it skips a result.json it cannot parse, and now it also skips one that is not an object. It used to count that as an unrecorded execution and refuse the whole batch. Its comment names where such a cell of this run is reported: run_cell, or _unsupported_cell.
• dokimi/tests/test_repo_run_unreadable_result.py (new; tests/test_repo_run.py is at the 1000-line cap):
  • A three-rep batch where r1's result.json is cut in half, a JSON list or not UTF-8, r2 is finished and r3 is missing. The rerun logs r1 by path, reuses r2, runs r3 (one agent started), prints 1 ran, 1 already done, 1 crashed, exits 1 and leaves r1's file byte for byte.
  • A task this host cannot run, with and without --force. One finished cell's result.json is [] and the other is cut in half. Both cells are logged by path, the run exits 1, both files are unchanged and no unsupported.json is written. The test reuses test_repo_host's fake host probe.
• dokimi/docs/repo-evaluation-contract.md: the paragraph on rerunning a batch gains sentences on both cases.
• Changelog fragment (Fixed).

Risk

Only a batch with a result.json that is not a readable JSON object behaves differently. Its other cells now run instead of the command stopping. A non-object result.json outside this run's plan no longer refuses the batch as unrecorded; it is skipped, as an unparsable one already was. A kept cell of a task this host cannot run, whose result.json cannot be read, now fails the run (exit 1) where main counted it as already done and could exit 0. Every case in the tables above exits 1 on this branch. A readable result is reused or refused exactly as before.

Testing

On a Linux review host, at 84c92bb (on main e97ff7a):

• make ci-local: 30 passed, 0 failed, 7 skipped (external-catalog-smoke, which needs Docker; test-latest, docs.yml:deploy and pip-audit.yml:report, whose job conditions are false on a PR; cli-surface / Package the result; conda-forge-deps, no micromamba; semgrep secrets, offline). 10656 tests passed and 4 were skipped; changed-line coverage was 100% (32 lines measured). The matrix legs on other Pythons (3.11, 3.12, 3.14) were not run.
• The repros above, through the CLI with the repo's fake Docker runner, fake harness and fake host probe. On main the batch ends with the bare error, or counts the unsupported task's cells as done; here each cell is reported and the batch continues.
• The new tests, run against main's code: all 5 cases fail. The unsupported-task test also fails, in both cases, against the first commit alone (02a2640).
• Mutants, each run against the new tests on a copy of the commit that added the code:
  • check_batch_executions back to a bare json.loads (a list counts as unrecorded): the list case fails.
  • _finished_result returns a non-object: the list case fails.
  • UnreadableResult a ValueError instead of a PrepError (not reported per cell): all 3 cases fail.
  • The UTF-8 branch dropped: the UTF-8 case fails.
  • _unsupported_cell back to counting any existing result.json as done, without reading it: both unsupported-task cases fail (no exit 1).
  • _unsupported_cell reads the file with run_cell's hint (--force replaces it): both unsupported-task cases fail.
  • run_batch's handler catches only OSError, so UnreadableResult ends the batch: both unsupported-task cases fail.
• Merged together with the other follow-up branch (test/leftovers-2026-10-30) on main e97ff7a, at 84c92bb: pytest -q -n auto 10658 passed, 4 skipped; mypy and ruff check . clean.

Revert

A plain revert of both commits. Nothing is stored or versioned by this change.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[Copilot]

🟡 Changes recommended

Corrupt retained results can be counted as completed on hosts that lack required CPUs, bypassing validation and per-cell error reporting even with --force.

1 open issue

• Validate retained results for unsupported tasks · New

What changed in this PR

1 new finding.

This follow-up to #238 makes malformed finished-cell results report per-cell errors instead of stopping an entire repo run batch.

Changes:

• Adds shared result parsing for UTF-8, JSON syntax, and object-shape validation.
• Adds regression tests covering error reporting, continued execution, and preservation of corrupt files.
• Updates the rerun contract and changelog.

File	Description
dokimi/​tests/​test_repo_run_unreadable_result.py	Tests malformed results alongside reused and unfinished cells.
dokimi/​dokimi/​repos/​runner.py	Adds result validation and integrates it with batch checks and cell reuse.
dokimi/​docs/​repo-evaluation-contract.md	Documents per-cell errors and recovery options.
dokimi/​changelog.d/​20261030_120000_antonwolfy_repo-run-unreadable-result.md	Records the fix and file-preservation behavior.

_{Review effort: Balanced}

[antonwolfy]

Fixed in 84c92bb, as a new commit on top. The unsupported-task branch of run_batch.one moves into a helper, _unsupported_cell. When the cell has a result.json, the helper reads it through _finished_result. On UnreadableResult (or an OSError reading it), one() records the cell in state["errors"] and logs ✗ <cell>: <path>: ... the same way the PrepError handler does. The file stays as it is, and the cell is neither rerun nor marked unsupported. A readable kept result is still counted as already done.

On this path --force keeps the file, so the message's recovery hint differs from run_cell's: "this host cannot run the task, so --force keeps the file; remove it, and the cell runs on a host that meets the task's requirements". The comment on the check_batch_executions catch now says the cell is reported in run_cell, or in _unsupported_cell for a task this host cannot run.

New test: test_an_unreadable_kept_result_of_a_task_this_host_cannot_run_is_that_cells_error, with and without --force. Both fix_race cells finish on a capable host. Then r1's result.json is replaced with [] and r2's is cut in half, and the batch is rerun on a 2-CPU AMD host. Both cells are reported by path, the run exits 1, both files are unchanged, and no unsupported.json is written. The test fails if the branch goes back to counting any existing file as done.

The contract paragraph and the changelog fragment now cover this case.

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s).

Review details

• Commit reviewed: 84c92bb against merge base e97ff7a
• Coverage: 4 files in 1 groups
• Engine: OpenCodeReview v1.13.3, us.openai.gpt-6.1-sol
• Usage: 106k tokens (60k from cache), 23s
• Tool calls: 9 (7 file_read, 2 code_search)
• Repository runner unreadable-result handling, tests, and documentation: 4 files

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.55%. Comparing base (e97ff7a) to head (84c92bb).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #304   +/-   ##
=======================================
  Coverage   93.54%   93.55%           
=======================================
  Files         275      275           
  Lines       35701    35726   +25     
  Branches     6053     6055    +2     
=======================================
+ Hits        33398    33423   +25     
  Misses       1486     1486           
  Partials      817      817           

Files with missing lines	Coverage Δ
dokimi/dokimi/repos/runner.py	95.59% <100.00%> (+0.32%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.