Repository navigation
repo tasks: authoring replay tasks from merged PRs #294
Description
Problem
repo tasks replay builds dokimi:review_replay tasks, which measure what guidance does to an agent reviewing code. Most guidance is read while writing code. Of the eight AGENTS.md files in IntelPython/dpnp#3099, most lines say where a new function's kernel and dispatch helpers go, how tests are laid out, and what a changelog entry looks like. The review replay of that PR exercised little of it.
dokimi can already grade such work. A task.json with family: feature or fix and the repository's own grade entry point does it (contract §3). But those tasks are written by hand, so this measurement is skipped too, just as review counting was before repo tasks replay.
Proposal
Build authoring tasks from merged PRs, with the same input and the same rules for what to leave out as review replay: repo tasks replay --kind authoring, or a sibling subcommand. Per merged PR:
- Base: the merge base of the PR with
mergeCommit^1, as for review replay. - Prompt: the PR's title and description, with links to its commits and diff and anything quoting the change removed.
--prompt-filereplaces the default. - Hidden tests: the test files the PR added or changed. They are held out of the workspace and applied only at grade time.
- Grade: the repository's test command from the manifest, on the hidden tests; the gate is that they pass.
oracleis the PR's own diff, sorepo tasks checkcan show that the base fails and the oracle passes. - Left out: guidance files and
eval_dir, as review replay leaves them out.
Open questions
- Which PRs qualify: with tests, under a size limit, not only refactors.
- Leaked tests: a description that names the test functions.
- Build cost: dpnp needs its SYCL extensions built for every cell. How much can the manifest's caches and a prebuilt base image save?
- Partial credit: some hidden tests passing. Should it count, as a score rather than a gate?
- Flaky tests: run the oracle twice in
tasks check?
Done when
On a window of merged PRs of a real repository, the command writes tasks that repo tasks check accepts (the untouched base fails, the oracle passes). The tests use a fixture repository with no network.
Activity
napetrov commentedon Nov 1, 2026
A second source for replay tasks (#342): agent failures, not only review threads. A ledger entry is commit + prompt + check, and it becomes a
dokimi:failure_replaytask. jimy-r's incident ledger is a possible format. #336'srepo_ledger.v1already links rules to cases and could reference these.antonwolfy commentedon Nov 1, 2026
Proposed answers to the open questions, and the scope after napetrov's comment above.
repo tasks checkdecides. A task whose base already passes its hidden tests fails it (§4.3: the untouched workspace must fail the gate), so a refactor drops out there without a heuristic. The build's own filter only needs to be cheap: the PR adds or changes a test file, and its non-test diff is under--max-lines(default 400).task.jsonrecords how many it dropped. A PR with more than a few such lines (say 5) is skipped as one whose description is its tests.grade.jsonalso carries the passing share as a metric, sorepo compare --metriccan read it.repo tasks checkruns the base and the oracle twice each. A task whose two runs differ is reported flaky and not counted as ok.public_at; here the PR'screatedAt), sorepo compareflags both kinds by one rule.Order. This is the last of #290–#294. It comes after #290's recording PR, and after #325, which moves
repo tasks checktoeval checkand edits therepo tasksparser this issue adds to.feat(repo): repo tasks authoring builds hidden-test tasks from merged…