[antonwolfy]

Summary

How much of a repository's past review its guidance would have caught is counted by hand today, and #260 lists the ways that count went wrong. dokimi:review_replay already measures it, but its gold.json and task directories are written by hand. dokimi repo tasks replay writes them from a threads file that gh api graphql wrote: one task per merged pull request, holding the change as it was reviewed and the reviewers' line threads as gold.

gh api graphql --paginate --slurp -F query=@replay-threads.graphql \
    -F q='repo:IntelPython/dpnp is:pr is:merged merged:2026-10-01..2026-10-27' > threads.json
python cli.py repo tasks replay https://github.com/IntelPython/dpnp@master --threads threads.json \
    --out .agent-eval/tasks --manifest-file dpnp-eval.yaml

Closes #260.

Changes

• repo tasks replay <source> --threads FILE --out DIR [--window N] [--prompt-file F] [--force], next to tasks check in commands/repo_run.py. The code is in a new package, dokimi/repos/replay/:
  • threads.py reads any JSON that holds pull request objects with number and reviewThreads: one response, the pages --paginate --slurp writes, or a list. A commit that is not a full 40- or 64-character hex SHA counts as missing, so no value from the file reaches git as an option or a revision expression.
  • A thread is gold when its first comment was written by someone other than the PR's author, not by a bot (__typename Bot, or a login ending in [bot]), on a line of the new file (subjectType other than FILE, diffSide other than LEFT, an originalLine), with its originalCommit. Every other thread, and every non-empty review body by a reviewer, is printed with the reason.
  • build.py writes one directory per PR:
    • The reviewed revision is the originalCommit most gold threads were written on; on a tie, the one with the latest comment.
    • base is the merge base with mergeCommit^1.
    • change.patch is the diff from base to the reviewed revision as one git am mail, titled with the PR's title. git am strips the CR at the end of each line of a plain mail, so a diff holding a CR (a file with CRLF line ends, such as a .bat) is written base64-encoded, which keeps it; any other patch stays readable.
    • task.json sets grader: dokimi:review_replay, score: {metric: recall, max: 1.0}, window and the prompt, and adds source_pr, source_url, reviewed_commit, merge_commit and left_out_files.
    • gold.json holds {id, file, line, kind: unknown, comment, addressed, reply, url} per thread.
    • The layout follows oneDAL's replay tasks: change.patch, gold, source_pr, reviewed_commit, and ids such as pr3580-t1.
  • history.py asks git through plumbing (diff-tree), so a porcelain setting such as diff.noprefix cannot change a patch. It also holds the -U0 hunk arithmetic and the first-parent walks.
• The two decisions from the issue comment:
  • Files the guidance globs match, and files under eval_dir, are left out of the patch, and their threads out of the gold. A PR with no other change gets no task.
  • addressed comes from walking first parents from the thread's originalCommit to headRefOid. A commit that changes the file within window lines of the carried line addresses the thread; deleting the file is such a change. The commit must have one parent, and no bot may have written it ([bot] in the author's name or email, as in pre-commit.ci's fixes).
    • Any other commit only moves the line. One that rewrites the line ends the walk, and the thread is printed for a maintainer to check.
    • When no first-parent path exists (a force-push, or the revision is reached only through a merge), the diff to the head is read instead, and the thread is printed too.
  • One difference from the comment: commits that mergeCommit^1 holds are not excluded. The base branch's commits reach the PR only through merges, which the walk already treats as moves. Excluding them would also drop the author's own fix in one case: a maintainer merges the PR branch into main partway through review.
• GitHub's output differs from what the issue assumed in two places:
  • originalCommit belongs to each comment, not to the thread.
  • A thread's path is the file's latest name. The name at the thread's revision is therefore followed back through renames with git log --follow. On dpnp PR 2841, sparse/_base.py was edited and then renamed to _lib/_sparse.py, so one diff over the range shows a deletion and an addition. A thread from another revision is carried to the reviewed one commit by commit, for the same reason.
• A remote source's clone has no refs/pull/*, so the commits it lacks are fetched from origin by SHA. A local repository is read as it is; the docs give the refspec to fetch first.
• A task directory that already has a task.json is kept, with the maintainer's edits, unless --force.
• Exit codes: the command exits 1 when no pull request gave a task. It also exits 1 on an error that is not one pull request's own (the threads file, the source, the manifest, the prompt file, writing a task), printing ✗ and the reason. A git failure while one pull request is built skips only that pull request, with the reason.
• Docs: a new §4.6 in the contract (the command, the gh query, the rules), a §9 example, architecture.md, and a §6 bullet in preparing-a-repository.md. Changelog fragment (Added).

The issue's open questions:

• A thread on a revision the PR later rewrote: it is anchored on originalLine at its originalCommit and carried to the reviewed revision. It is printed and left out when its line changed on the way, or when its file is in no revision.
• A PR that merges main in after review: addressed comes from the PR's own commits, as above, not from the merge commit.
• A declining reply does not override addressed. reply is in the gold for the maintainer.
• One task per PR, or per revision: one per PR. On dpnp PR 2841 (85 threads over several review rounds), many threads from other rounds are left out because their line changed before the reviewed revision. A task per reviewed revision would keep them; that is not done here.

Risk

A new command, so no existing behaviour changes. It writes only under --out. For a remote source it also fetches objects into dokimi's own cache clone. A local repository's git directory is never written to. A PR title that starts with a bracketed tag ([WIP] …) loses the tag in the replayed commit, since git am strips bracketed prefixes from the subject.

Testing

On a Linux review host at 80382d0 (on main 3a1f287):

• make ci-local: 28 passed, 0 failed, 7 skipped (the Docker catalog smoke, test-latest, the cli-surface packaging step, conda-forge-deps without micromamba, docs deploy, the pip-audit report, semgrep). 9,343 tests passed, 4 skipped; changed-line coverage 98% (455 lines).

• tests/test_repo_replay.py: 44 tests on a toy history built per test, with threads files in the shape --paginate --slurp writes:

  • PR 7 is a squash-merged PR that merges main in after review, renames a file, then addresses some threads. It covers gold vs left-out threads and every reason, the reviewed revision, base, the patch and the left-out files.
  • addressed:
    • a change by the author;
    • a change main brought next to the line, which does not count, while the diff to the merge commit has it within the window;
    • a deleted file;
    • a file edited and renamed;
    • a merge that rewrites the line (printed);
    • a pre-commit.ci commit next to the line, which does not count;
    • --window.
  • Files:
    • a binary file;
    • a file with CRLF line ends (a base64 patch, applied under guidance:none with its CRs);
    • the last line of a file with no final newline;
    • a path with glob characters next to a file the glob would match.
  • Carrying a line between revisions, forward and backward, also across a rename that one diff does not show.
  • The PRs that get no task: guidance only, not merged, no head, no thread, a merge commit not in the source, a reviewed revision already on main, no gold thread left.
  • A git failure in one PR, which skips that PR while the others are written. Also a force-pushed PR (flagged) and a PR merged with a merge commit (base is its first parent).
  • Kept and --force, --prompt-file.
  • A file:// remote source that has to fetch the PR's commits by SHA, and gives the same task as a local source.
  • The input shapes and SHA validation, every way an input fails (exit 1 and the message), and the hunk arithmetic.
  • The generated task passes repo tasks check (oracle recall 1.0, null 0.0) and prepares under guidance:none and guidance:raw.

• 38 mutants of the new code, each failing a test.

• An independent review pass before opening found:

  • the CRLF patch that git am rejected;
  • -f instead of -F in the documented gh call;
  • bot commits counted as responses;
  • a silent stop when a merge rewrote the line;
  • one git failure aborting the whole run;
  • doc and test gaps.

  All are fixed in this commit.

• Real data: IntelPython/dpnp, 47 PRs merged 2026-10-01..27, read with the §4.6 query.

  • Results: 15 tasks, 90 gold threads, 54 of them addressed; the other 32 PRs had no line thread by a reviewer.
  • repo tasks check on all 15 tasks: 15/15 ok.
  • Each task prepares under guidance:none and guidance:raw.
  • 2.5 minutes, including a full clone of dpnp.

• repo run on the 15 tasks, us.anthropic.claude-sonnet-5-5, 1 rep, Claude Code 2.1.312 in a python:3.12-slim image with git. 45 cells, none failed, $3.37 in all (3 to 13 turns per cell).

  • Arms: guidance:none, guidance:raw, and guidance:rev:0cf6730, the head of Add AGENTS.md guidance for coding agents IntelPython/dpnp#3099, which adds 8 AGENTS.md files.

  • dpnp's master holds no guidance file, so none and raw are the same workspace, and the gap between them is the noise at one rep.

  • repo compare --baseline guidance:none --metric recall_addressed (10 tasks have an addressed thread; the other 5 have none and are unpaired):

    arm	recall (15)	recall_addressed (10)
    guidance:none	0.462	0.445
    guidance:raw (same workspace)	0.467	0.550
    guidance:rev:0cf6730	0.497	0.281 (-0.164 [-0.503, +0.176], p 0.375)

  • This shows the measurement runs end to end. It says nothing yet about the guidance: the difference between two identical arms is as large as the difference the guidance makes, so a reading needs more reps and more PRs.

Revert

Revert the commit. Nothing else reads the files it writes.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 96.73913% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.10%. Comparing base (3a1f287) to head (18bdeb1).
⚠️ Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/repos/replay/history.py	93.54%	4 Missing and 6 partials ⚠️
dokimi/dokimi/repos/replay/build.py	97.93%	1 Missing and 2 partials ⚠️
dokimi/dokimi/repos/replay/threads.py	98.27%	1 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #265      +/-   ##
==========================================
+ Coverage   92.83%   93.10%   +0.27%     
==========================================
  Files         262      269       +7     
  Lines       33016    33923     +907     
  Branches     5510     5692     +182     
==========================================
+ Hits        30649    31583     +934     
+ Misses       1571     1536      -35     
- Partials      796      804       +8     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/repo_run.py	93.76% <100.00%> (+0.73%)	⬆️
dokimi/dokimi/repos/replay/__init__.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/replay/threads.py	98.27% <98.27%> (ø)
dokimi/dokimi/repos/replay/build.py	97.93% <97.93%> (ø)
dokimi/dokimi/repos/replay/history.py	93.54% <93.54%> (ø)

... and 23 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

+ added command `dokimi repo tasks replay`
    + argument `source`
    + option `--cache-dir`
    + option `--force`
    + option `--help/-h`
    + option `--manifest`
    + option `--manifest-file`
    + option `--out`
    + option `--prompt-file`
    + option `--threads`
    + option `--window`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

[xaleryb]

• Ran: the test suite, ruff, mypy and bandit on this head merged with current main, plus a no-op mutation of every new branch.
• Tests (py3.14): [nested-too-deep] relies on json.loads raising RecursionError. On 3.14 that depends on the C stack size, so it parses on the runner and the job is red. b935366 stubs the parser to raise instead, so the except branch stays pinned.
• PR merged by fast-forward (mergeCommit == headRefOid) -> task written with base = the PR's first commit, so the patch lacks it -> expected: skip with a reason. Fixed in 423339a, and §4.6 now says so.
• Thread on an earlier commit, answered by a later commit that the reviewed revision contains -> in gold with addressed: true, though the workspace already has the fix -> expected: left out. Fixed in 6f944b6.
• 572a0a9 and 18bdeb1 add tests for the majority rule in _reviewed and for _oid's full-SHA match.
• For the author: 6f944b6 drops the thread (printing a note) instead of keeping it, and 423339a also skips single-commit fast-forwards. Please confirm both choices.

[xaleryb]

Approved at 18bdeb1: every point from our review is fixed at this head or left as a named follow-up, the full suite, ruff, mypy and bandit pass on Linux, and CI is green.