[antonwolfy]

Summary

Found while evaluating IntelPython/dpnp#3099 (AGENTS.md guidance) with dokimi: 26 review_replay tasks, two arms (guidance:none and guidance:rev:<sha>), 3 reps, Claude Code, 156 cells. Reading that batch with repo compare ran into three gaps.

What the reader needed	What main gives
Whether the guidance arm delivered its files	Nothing per cell. The only evidence was the size of each run's first prompt, read by hand from every agent.stdout: median 19,380 tokens against 16,315 for guidance:none (each ±20 over 78 cells), +3,065. That matches the root AGENTS.md (7,015 chars) and none of the 7 nested files (21,199 chars), which load only lazily. result.json agent has whole-run token totals only, and repos/events.py reads the assistant messages but not their usage.
The 17 tasks reviewed before the first guidance commit vs the 9 after	repo run has --task; repo compare has none, so every statistic of the split was recomputed by hand.
How to read recall +0.022 [-0.066, +0.100], p 0.62, 26 tasks	Just the row. The report wrote "no measurable recall gain". With SD 0.222 for the per-task deltas, 80% power at a two-sided 0.05 detects only about ±0.12, (1.96 + 0.84) * 0.222 / sqrt(26): the result is inconclusive.
How to read an interval and a p that disagree	Just the row. metric:n_findings +0.89, t interval [-0.091, +1.860], sign-flip p 0.0497: the report quoted "p = 0.049" for a descriptive row outside Holm's family, one of about a dozen comparisons. The score row over the 6 post-cut tasks no guidance commit named: +0.139, bootstrap [+0.028, +0.250], which excludes 0, while the exact sign-flip p is 0.25 (three non-zero deltas, all positive, so p cannot go below 0.25).

Related: #292 asks for an interval that agrees with the sign-flip p by construction; the disagreement note here is the stopgap until then. #290 asks repo compare to flag tasks a guidance revision may have seen; --task is how the rest get compared on their own.

Changes

• repos/events.py: Transcript.first_prompt_tokens, the input + cache_creation + cache_read tokens of the first main-thread call (parent_tool_use_id null) of a Claude Code stream-json trace. It is read with dokimi.context.trace.parse_trace, so a call is counted once by message id and its blocks merge as context trace does; there is no second implementation. null when the trace has no such call, and for the text format.
• repos/runner.py: result.json agent.first_prompt_tokens (null until set, like the other token fields).
• repos/compare.py: first_prompt_tokens is a descriptive measure next to cost_usd and turns (per-task mean over the cells that have it). Like a process count it is compared only when both arms report it for some shared task: a batch written before the field has no row, not a row of zeros, and when only one arm has it a warning names it.
• repo compare --task ID, repeatable (commands/repo_run.py, compare_tasks.select_tasks): compares only those task ids. An id no finished cell of the batches has is refused (exit 1, ✗ no finished cell of the compared batches has --task t9). The tasks left out are not listed as unpaired and their cells not as missing. A missing cell's task is the one its meta.json records; a cell only plan.json names gets the longest task id of the batch's finished cells that its name starts with followed by __, and a cell neither gives a task for is missing only without --task. --json records the selection as task_filter ({"tasks": [...], "of": <tasks with a finished cell>, "dropped_arms": [...]}, null without --task): without --arm, dropped_arms names the arms with no finished cell among the selection, which drop out of the comparison. The text header prints tasks: N of M (--task) and, when an arm dropped out, not compared, no finished cell among the --task selection: guidance:raw.
• Reading notes under each comparison (compare._reading_notes), after the changed: and headroom notes. No delta, interval, p or table column changes:
  • a gate or score row of 2+ tasks whose interval includes 0: score inconclusive: the interval includes 0; an uncorrected t-test estimate puts the effect 80% power detects at this spread at about ±0.676 (6 tasks), before Holm and not for the sign-flip test. mde = (t(0.975, n-1) + t(0.80, n-1)) * sd / sqrt(n) over the per-task deltas (sample SD), a planning figure (an approximate, uncorrected t-based estimate, not the power of the sign-flip test nor of the Holm-corrected decision), is in every --json row, null for one task or deltas with no spread. It is null too where the sign-flip p cannot reach 0.05: over k changed tasks it cannot go below 2 / 2**k, so at 5 or fewer, and the note says so instead: gate inconclusive: the interval includes 0; with 2 changed tasks the sign-flip p cannot go below 0.5: no effect can reach p < 0.05 here. A gate or score row is decided by p_holm, so its reach is read after Holm's correction (compare_reach.holm_floor, with correct_family): p_holm_floor is the row's p_holm with its own p put at that floor and the family's other p as observed, and where it is not below 0.05, mde is null and the note names Holm: two score treatments of six changed tasks each print score inconclusive: the interval includes 0; with Holm over 2 rows, p_holm cannot go below 0.0625 here: no effect can reach p_holm < 0.05. p_holm_floor decides only whether mde is given, not its size. A family of one row is the raw rule; descriptive rows keep it;
  • a row whose interval and p disagree gets interval_p_disagree: true and a note. For a gate or score row the p is p_holm, the one that decides it, set once Holm's correction has run. On the 6-delta case above it prints score: the interval [+0.028, +0.250] excludes 0 but p_holm 0.25 is ≥ 0.05; they come from two methods, a percentile bootstrap, too narrow over few tasks, and a sign-flip test; read p_holm. A descriptive row compares its raw p, names the t interval instead and does not say "read p_holm". On a gate or score row that has this note, the inconclusive note is not printed too;
  • descriptive rows with p < 0.05 are named on one line: p < 0.05 on turns: descriptive rows, their p uncorrected and outside Holm's family, which covers gate and score only.
• Docs: contract §4 (agent.first_prompt_tokens) and §9 (the measure, --task, the three notes, mde and p_holm_floor), a line in docs/context-window.md, repos/compare_tasks.py and repos/compare_reach.py in docs/architecture.md, and the compare.py module docstring.
• docs/reference/cli-interface.json, the CLI snapshot main's test(cli): flag registry, ratchet contract test, CLI snapshot (#94 F1a) #254 checks, regenerated with make cli-snapshot for repo compare --task. The flag needs no allowlist row: --task is already a canonical spelling of the inputs concept in dokimi/cli_flags.py (ID, text, repeatable, "a filter on --tasks").
• Tests: new tests/test_repo_compare_reading.py (test_repo_compare.py is 955 lines, near the 1000-line limit for a test file), one trace test in tests/test_repo_reach.py, and the agent-record tests in tests/test_repo_run.py. The fake harness in tests/repo_eval_fixture.py puts usage on its FAKE_USAGE assistant message.
• Changelog fragment (Added).

Risk

• result.json agent gains one key; readers that take the dict as it is see one more field. No schema covers this file.
• repo compare --json rows gain mde, p_holm_floor and interval_p_disagree, and the summary gains task_filter (tasks, of, dropped_arms). Every existing key and value is unchanged.
• The text output gains note lines under some comparisons, and a first_prompt_tokens row for batches run from this change on. Older batches print as before apart from the notes.
• repo run parses each Claude Code trace a second time (in context.trace) to get the first call. That is linear in the trace size and runs once per cell.
• repos/compare.py grows from 696 to 800 lines (limit 800); the --task selection and CompareError are in the new repos/compare_tasks.py (52 lines), min_detectable, ALPHA, POWER and the Holm floor in the new repos/compare_reach.py (32 lines), and dokimi.repos.compare.CompareError and dokimi.repos.compare.min_detectable still import. Open PR fix(model): one same-model rule for baseline, self-judge and repo compare #273 also edits compare.py. This change edits only the parts above and reformats nothing around them; merged together, the file will need a further split to stay under the limit.

Testing

On a Linux review host, at 30030ed (on main e966422):

• make ci-local: 27 passed, 1 failed, 7 skipped. Skipped: external-catalog-smoke (needs docker), test-latest, docs deploy and pip-audit report (job conditions false on a PR), cli-surface packaging (unknown expression locally), conda-forge-deps (no micromamba), semgrep secrets (not installed). The failure is pip-audit: Found 10 known vulnerabilities in filelock 3.32.7 (fixes in 4.0.9 / 4.2.1). That version is pinned in main's lock, which this branch does not touch; ci(deps): document and ignore ten more filelock 3.32.7 advisories #288 documents and ignores those advisories. 10196 tests passed and 4 were skipped; changed-line coverage 100% (42 lines). The cli-surface step reports the one interface change: dokimi repo compare, added option --task.
• Touched and related test files (test_repo_compare_reading.py, test_repo_compare.py, test_repo_compare_paths.py, test_repo_reach.py, test_repo_run.py, test_repo_process_counts.py, test_repo_uptake.py, test_repo_skill_arms.py, test_context_trace.py): 456 passed. Guard tests (test_docs_code_refs.py, test_growth_ratchet.py, test_audit_test_quality.py, test_import_architecture.py): 290 passed. ruff check on the changed files and mypy on the whole package (278 files) are clean.
• At main, the new tests fail: test_repo_compare_reading.py stops at import (min_detectable does not exist), and test_first_prompt_tokens_is_the_first_main_thread_call_counted_once and three test_repo_run.py agent-record tests fail. All of them pass here.
• Mutants: each was run on a copy of the commit against the test that pins it, and all 17 failed. An unmutated copy passed (179 tests). The mutants:
  • the trace read from every call instead of the main thread's;
  • context.trace's block merge dropped;
  • the runner not writing the field, or not defaulting it to null;
  • first_prompt_tokens dropped from _task_values, or from the both-arms rule;
  • for --task: the unknown-id check off, every cell kept, missing cells not filtered, the header line dropped, the CLI not passing the flag;
  • Z_POWER set to 0, or stdev swapped for pstdev;
  • the inconclusive condition inverted;
  • the disagreement test flipped;
  • "read p" dropped;
  • the descriptive-p test flipped.
• Not run: a real repo run against a model, since this host makes no model calls. The dpnp numbers above come from that evaluation's batch, which is not on this host.
• main merged in after fix(cli): run, compare, arms, tasks, explore and repo exit with a message on bad input or output #238 (cad52dd). The conflicts were in cmd_repo_compare, which now passes --task and also catches fix(cli): run, compare, arms, tasks, explore and repo exit with a message on bad input or output #238's CellFileError, and in _task_values, where the first_prompt_tokens list moved into fix(cli): run, compare, arms, tasks, explore and repo exit with a message on bad input or output #238's reading_cells block. compare.py keeps import math for min_detectable and is 770 lines. On that commit, the compare, run, reach and context-trace tests with the guard tests: 644 passed; ruff and mypy are clean on the merged files.
• Merged together on main f0f95a2 with the other PRs from this evaluation (fix(repo): scan says why it is incomplete #295 at 4fa7783, feat(repo): compare records the start context, filters tasks, and flags inconclusive rows #296 at cad52dd, feat(repo): replay grades record each credited hit; repo report --hits lists them #297 at 0a7539a, fix(novelty): a capped run says how far the cap reached #299 at fa453d8), on a workstation: the full suite has 10482 passed and 57 skipped, and the same 5 failures and 1 collection error as main alone there (10456 passed). All six come from that machine's setup: no docker, no hypothesis, and two tests that depend on its git and shell settings.
• Review round, 00f9b40 (test only): test_first_prompt_tokens_of_an_older_batch_is_no_row_not_zero now also checks a batch where only the baseline reports first_prompt_tokens, with the same warning and no unpaired entries. test_first_prompt_tokens_is_the_first_main_thread_call_counted_once gives the main call a third, sparser usage line after the full one. Before the change, two mutants passed these tests: the one-arm skip checking only the baseline arm (if not has[0]), and "last line wins" in context.trace's merge. Each now fails, and "first line wins" fails as well: 3 of 3 caught. An unmutated copy passes both files (100 tests). make ci-local: 25 passed, 3 failed, 7 skipped. 10528 tests passed and 4 were skipped; changed-line coverage 100% (43 lines). Two of the failures are gitleaks: its download got HTTP 504 four times, so Scan commits had no binary (exit 127). The third is pip-audit, the same filelock 3.32.7 finding as before; that version comes from main's lock. The skips are the same 7 as before.
• Review round, eb5fbce, 5eff69d, aeea5f1, d986407, 68b4c86, on a Linux review host, at 68b4c86 (on main f0f95a2). eb5fbce moves the --task selection, without behaviour change, into the new repos/compare_tasks.py so compare.py stays under 800 lines (799 now). 5eff69d: mde is null where the sign-flip p cannot reach 0.05 (2 / 2**k ≥ 0.05, at 5 or fewer changed tasks), and the note says how low p can go. aeea5f1: a gate or score row's interval_p_disagree and its note read p_holm. d986407: a missing cell's task comes from its meta.json, else the longest known task id its name starts with. 68b4c86: task_filter.dropped_arms and a header line name the arms --task leaves without a cell. New tests in test_repo_compare_reading.py: test_no_mde_where_the_sign_flip_p_cannot_reach_005 (the four-task score row, and five changed of six), test_a_gate_row_of_two_changed_tasks_has_no_mde ([1, 1, 0]), test_a_gate_interval_against_p_holm_is_flagged, test_an_interval_across_zero_against_a_raw_p_below_005_is_not_flagged_when_p_holm_is_not (bootstrap_ci replaced by one spanning 0: in 7,975 random score rows with a sign-flip p in [0.025, 0.05), no bootstrap interval included 0), test_task_reads_a_missing_cell_s_task_from_its_meta_or_the_known_ids (model m__v1, arm skill:lint__v2, tasks fix, fix__a, fix__guidance-raw), test_task_names_an_arm_that_ran_none_of_the_selected_tasks. test_an_interval_across_zero_is_inconclusive_with_its_mde now uses six changed tasks. At 525e9d2 the new and changed tests fail (11 of 19 in the file) and here they pass. Mutants, each on a copy of 68b4c86 against the tests that pin it: 13 of 13 failed (the floor check dropped, counted over all tasks, doubled; the floor note dropped; the Holm flag not set, or set from the raw p; the note quoting the raw p; meta.json ignored, the plan fallback dropped, the shortest id taken; dropped_arms always empty, filled under --arm, not printed). An unmutated copy passes (145 tests). The fixes from the colleague's commits hold: with the prefix match put back (349d544) or the --task scope suffix removed (d5f2d7c), their tests fail. make ci-local: 27 passed, 1 failed, 7 skipped. 10538 tests passed and 4 were skipped; changed-line coverage 100% (100 lines). The failure is pip-audit, the same filelock 3.32.7 finding as before (10 known vulnerabilities, 3 ignored); main ignores those advisories since ci(deps): document and ignore ten more filelock 3.32.7 advisories #288, which this branch predates, and CI tests the merge with main. The skips are the same 7 as before.
• Review round, c19f9d2, 63c112e, on a Linux review host, at 63c112e (on main 83f18fb, merged into the branch at 8fa6756). c19f9d2 regenerates docs/reference/cli-interface.json for repo compare --task: at 8fa6756, test_the_cli_interface_snapshot_is_current (test(cli): flag registry, ratchet contract test, CLI snapshot (#94 F1a) #254) failed on the stale snapshot and passes now. --task needed no allowlist row, being a canonical spelling of the registry's inputs concept, and ci: fail comment blocks over two lines that a PR adds or edits #298's check_comment_blocks.py --base origin/main --head HEAD finds no comment block over two lines in the branch. 63c112e: a gate or score row's mde is null where Holm keeps its p_holm from 0.05 even with its p at the sign-flip floor (p_holm_floor, compare_reach.holm_floor), and the note names Holm. New tests in test_repo_compare_reading.py: test_no_mde_where_holm_over_the_family_keeps_p_holm_from_005 (two score treatments of six changed tasks each: both mde null, p_holm_floor 0.0625, the Holm note twice) and test_a_row_holm_still_lets_reach_005_keeps_its_mde (eight changed tasks, p_holm_floor 0.0156, keeps its mde beside a six-task row that loses it). At 8fa6756 both fail (an mde of 0.676 where null is due; no p_holm_floor) and here they pass. Mutants, each on a copy of 63c112e against that file: 5 of 5 failed (the mde reset dropped; the floor taken without the family; the floor not put in, the observed p used; the Holm note dropped; the family counted over every row). make ci-local: 28 passed, 2 failed, 7 skipped. 10661 tests passed and 4 were skipped; changed-line coverage 100% (112 lines); the CLI snapshot, comment-block and pip-audit steps pass. The two failures are gitleaks: its download got HTTP 504 four times, so Scan commits had no binary (exit 127). A gitleaks already on the host scanned the branch's commits instead: no leaks found. The skips: external-catalog-smoke (needs docker), test-latest, docs deploy and pip-audit report (job conditions false on a PR), cli-surface packaging (unknown expression locally), conda-forge-deps (no micromamba), semgrep secrets (not installed). Not yet run merged together with the other follow-up branches.
• Review round, fda7c07, on a Linux review host, at fda7c07 (on main 83f18fb). fda7c07 (docs and note text only; how mde is computed and every JSON key and value are unchanged) names mde for what it is: an approximate, uncorrected t-based planning estimate, the effect a one-sample t test at the uncorrected 5% level detects with 80% power at the row's spread, not the power of the sign-flip test nor of the Holm-corrected decision. The note now reads score inconclusive: the interval includes 0; an uncorrected t-test estimate puts the effect 80% power detects at this spread at about ±0.487 (8 tasks), before Holm and not for the sign-flip test. The compare.py module docstring, compare_reach.min_detectable, contract §9, docs/architecture.md and the changelog fragment say the same, and §9 says why no Holm-calibrated figure is given. Checked first at 63c112e: in the eight-task case of test_a_row_holm_still_lets_reach_005_keeps_its_mde, both raw p are 1.0 and the row's mde is 0.487, the one-row value, while Holm over the two rows passes a raw p of 0.024 (p_holm 0.048) and not 0.026 (0.052). Changed tests: test_an_interval_across_zero_is_inconclusive_with_its_mde and test_a_row_holm_still_lets_reach_005_keeps_its_mde assert the new wording. At 63c112e (the old wording) both fail and here they pass. Mutant, on a copy of fda7c07: "uncorrected" dropped from the note, both fail; 2 of 2 caught. compare.py is 800 lines (limit 800). make ci-local: 28 passed, 2 failed, 7 skipped. 10661 tests passed and 4 were skipped; changed-line coverage 100% (112 lines). The two failures are gitleaks: its download got HTTP 504, so Scan commits had no binary (exit 127). The installed gitleaks scanned the branch's commits instead: no leaks found. The skips: external-catalog-smoke (needs docker), test-latest, docs deploy and pip-audit report (job conditions false on a PR), cli-surface packaging (unknown expression locally), conda-forge-deps (no micromamba), semgrep secrets (not installed). Not yet run merged together with the other follow-up branches.

Revert

git revert of the commit. Cells written in the meantime keep agent.first_prompt_tokens in result.json; readers before this change ignore the key.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[github-actions]

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

~ `dokimi repo compare` parameters changed:
    + added option `--task`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

[antonwolfy]

@exai-bot /review

[exai-bot]

Note

AI-generated review of 30030ed1d020, requested by @antonwolfy, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
CI on this commit, at review start and at posting: 13 passed, 3 running, 2 skipped.
Model us.anthropic.claude-sonnet-5-5, effort default, 12 of 162 turns, 1.3 of 120 minutes, 528k in / 6k out tokens.

The change records first_prompt_tokens per cell and compares it as a descriptive measure, adds repo compare --task, and prints reading notes (mde, interval/p disagreement, uncorrected descriptive p) without changing any number. I read the diff against context/trace.py, compare.py and the test helpers and found no defects. The parse_trace reuse, the one-arm-reports skip rule, the task-filter handling of missing and unpaired, and the interval_p_disagree condition all behave as the description says. I could not run the tests, because pytest, numpy, ruff and mypy are not installed in the container and there is no network.

No issues found.

For a human reviewer

• Design call: the notes label a gate or score row inconclusive and give an mde computed from the observed spread of the per-task deltas. Confirm that framing is what you want users to read.
• Whether the interval/p disagreement note should fire on descriptive rows, since the t interval and the sign-flip p will often differ at small task counts and could add noise.

Repository checks

Each ran as the head of this pull request defines it.

Check	From	Result	Note
python scripts/check_growth.py	AGENTS.md	not run	no shell call ran this command
pytest tests/test_repo_compare_reading.py tests/test_repo_compare.py tests/test_import_architecture.py tests/test_repo_reach.py	AGENTS.md	not run	pytest is not installed in the container and there is no network
ruff check / mypy dokimi	.github/workflows/ci.yml	not run	tools are not installed in the container and there is no network

Not verified

I did not run the new tests (test_repo_compare_reading.py, test_repo_reach.py, test_repo_run.py) or the CI checks. I checked the bootstrap-interval expectations (lo > 0, hi = +0.250) against the code by reading only. python scripts/check_growth.py ran and printed only a note that origin/main has no growth baseline to compare against. I did not confirm compare.py stays under the complexity cap for _reading_notes.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.58%. Comparing base (83f18fb) to head (fda7c07).
⚠️ Report is 2 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #296      +/-   ##
==========================================
+ Coverage   93.52%   93.58%   +0.05%     
==========================================
  Files         274      277       +3     
  Lines       35625    35800     +175     
  Branches     6052     6070      +18     
==========================================
+ Hits        33320    33504     +184     
+ Misses       1489     1483       -6     
+ Partials      816      813       -3     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/repo_run.py	93.97% <100.00%> (+0.01%)	⬆️
dokimi/dokimi/repos/compare.py	99.77% <100.00%> (+0.29%)	⬆️
dokimi/dokimi/repos/compare_reach.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/compare_tasks.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/events.py	92.93% <100.00%> (+0.15%)	⬆️
dokimi/dokimi/repos/runner.py	95.26% <ø> (ø)

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[vchamarthi]

✅ OpenCodeReview: Review complete: 0 finding(s) across 11 selected item(s).

Review details

• Commit reviewed: 00f9b40 against merge base f0f95a2
• Coverage: 11 files in 3 groups
• Engine: OpenCodeReview v1.13.3, us.anthropic.claude-sonnet-5-5
• Usage: 950k tokens (792k from cache), 1m1s
• Tool calls: 43 (25 code_search, 16 file_read, 1 file_find, 1 file_read_diff)
• Docs and changelog for compare reading feature: 3 files
• Repo compare/events/runner/command implementation: 4 files
• Tests and fixtures for repo compare reading and run: 4 files

[antonwolfy]

Review round in 00f9b40, tests only. Both OpenCodeReview points held: a mutant survived each at cad52dd, and both are caught now.

• The one-arm skip of first_prompt_tokens is tested in both directions.
• The trace test pins the field-by-field merge of one message's usage lines.

On the two design points exai-bot left for a human reviewer:

• mde and "inconclusive". This is the framing intended. The evaluation that prompted this PR read recall +0.022 [-0.066, +0.100] over 26 tasks as "no measurable gain". At that spread, 80% power reaches only about ±0.12, so the honest reading is inconclusive. mde uses the observed spread of the per-task deltas, as a planning figure for the next run's task count. It is not a post-hoc power claim about this result, and the note prints only when the interval includes 0.
• The disagreement note on descriptive rows. It fires only when the interval and p fall on opposite sides of 0.05, not whenever they differ. That case is the one that misled the same report: n_findings +0.89, t interval [-0.091, +1.860], sign-flip p 0.0497, quoted as a finding. When the two agree, nothing is printed.

[vlad-perevezentsev]

Review at 00f9b40. The first_prompt_tokens record and --task are right. The reading notes can still mislead in two cases: mde at task counts where the sign-flip test cannot reach 0.05, and gate and score rows read against the raw p instead of p_holm. Both are in the line comments.

What I ran, at 00f9b40:

• make ci-local --offline: 17 checks passed, 0 failed, 13 skipped (network, docker and the other Pythons).
• paired_summary on small gate sets: [1, 0] gives mde 1.401 at p 1.0, and [1, 1, 0] gives 0.934 at p 0.5.
• compare with two treatments and 6 of 7 gate tasks up: p 0.031, p_holm 0.062, interval [+0.571, +1.000], and no note.
• compare --task fix on a batch with tasks fix and fix__a: a missing fix__a cell is still listed.

Checked, no issues

• Start context: _claude_stream_json takes the first main-thread call from parse_trace, so a message id's usage lines are merged field by field, as context trace does. A broken line goes into errors there and does not lose the cell's record. A killed cell writes null.
• Older batches: first_prompt_tokens goes through the same one-arm rule as the process counts. The tests now cover both directions and the sparse-later-line merge, as the resolved threads asked.
• Numbers: no existing value changes; mde and interval_p_disagree are new keys, and nothing in the schemas covers the compare JSON.
• Docs and changelog: the contract, context-window.md and the fragment describe the new fields and --task.

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 15 selected item(s).

Review details

• Commit reviewed: fda7c07 against merge base 83f18fb
• Coverage: 15 files in 2 groups
• Engine: OpenCodeReview v1.13.3, us.openai.gpt-6.1-sol
• Usage: 244k tokens (137k from cache), 42s
• Tool calls: 22 (17 file_read, 4 code_search, 1 file_read_diff)
• Repository comparison reading, reach analysis, and documentation: 10 files
• Repository run CLI and runner integration: 5 files

[Copilot]

🟡 Changes recommended

The detection-limit estimate misrepresents the exact sign-flip test’s small-sample capabilities, and ambiguous cell-name parsing silently omits unfinished cells when filtering by task.

2 open issues

• Missing-cell loader misparses task and arm names with underscores · New
• Finite t-based power estimate misstates exact sign-flip test limits · New

What changed in this PR

2 new must-fix findings.

This PR improves dokimi’s repository A/B comparisons by recording initial prompt size, supporting task subsets, and adding guidance for interpreting uncertain results.

Changes:

• Records and compares first-main-thread prompt tokens, while preserving missing values for older batches.
• Adds repeatable repo compare --task filtering with selection details in text and JSON.
• Adds detection-limit estimates and notes for inconclusive results, interval/p-value disagreements, and uncorrected descriptive comparisons.

File	Description
dokimi/​tests/​test_repo_run.py	Checks recorded prompt tokens and null defaults.
dokimi/​tests/​test_repo_reach.py	Tests first-call selection and merged usage records.
dokimi/​tests/​test_repo_compare_reading.py	Tests task filtering, token comparisons, and statistical notes.
dokimi/​tests/​repo_eval_fixture.py	Adds assistant-call usage to the fake harness.
dokimi/​dokimi/​repos/​runner.py	Persists first-prompt token counts.
dokimi/​dokimi/​repos/​events.py	Extracts initial prompt size through the shared trace parser.
dokimi/​dokimi/​repos/​compare.py	Implements task filtering, token comparisons, and reading notes.
dokimi/​dokimi/​commands/​repo_run.py	Exposes and forwards the repeatable --task option.
dokimi/​docs/​repo-evaluation-contract.md	Documents the new fields, filtering, and statistical guidance.
dokimi/​docs/​context-window.md	Documents first-prompt measurement in repository evaluations.
dokimi/​changelog.d/​20261029_150000_antonwolfy_compare-reading.md	Announces the new recording and comparison features.

_{Review effort: Balanced}

[napetrov]

Head at review: 00f9b40 (unchanged on fetch, 10-29). Pushed sha: 525e9d2.

Anton, the small fixes for Vlad's threads are on top of 00f9b40 and pushed as 525e9d2 (6 commits, signed off). Each fix has a test that fails on 00f9b40. Two of the threads are design questions, listed at the end, and are not changed.

Fixed, with the commits (pushed shas):

• compare.py _select_tasks, the --task prefix thread: a missing cell is matched to its task by splitting the cell name from the right (<task>__<arm>__<model>__r<rep>), not by prefix. fix no longer counts fix__a's unfinished cell. Test: test_task_matches_a_missing_cell_by_its_task_not_by_a_prefix. (349d544)
• Error text, the arm and metric thread: with --task, the baseline, treatment and --metric errors add "among the --task selection". Test: test_an_arm_or_metric_error_under_task_says_the_selection_is_the_reason. (d5f2d7c)
• The of count thread: I took the wording option. of is the number of tasks with a finished cell, the only ids --task can name. The contract now says so. Counting the planned tasks instead is a bigger change; say if you want it. (e00edbc)
• The mde nit (my review): min_detectable uses t quantiles at n-1 df, like the interval, so the note at 4 tasks says ±0.950, not ±0.639. The contract formula follows. Test: test_min_detectable_uses_the_t_quantiles_of_the_interval_at_small_n. (0c7d61f, 525e9d2)
• docs/context-window.md: the sentence now names the first prompt ("records the size of each cell's first prompt, in tokens, ..."). (5fc5198)

Not changed, for you to decide:

1. Sign-flip p vs mde (thread at compare.py:539). With k <= 5 changed tasks the exact p cannot go below 2/2^k, so no effect can reach p < 0.05, yet the note still names an mde. Options: (a) mde null and a note saying the sign-flip p cannot go below 2/2^k at this count; (b) keep mde and say it is not reachable by the test; (c) cap it. Recommendation: (a). It changes the mde JSON value at small k, and the module docstring says no number changes, so the docstring would need an update too.
2. Holm p in the disagreement check (thread at compare.py:540). interval_p_disagree is set before _holm, so gate and score use the raw p. Options: (a) compare with p_holm and print it in the gate and score note, as suggested; (b) keep raw p. Recommendation: (a), but it compares an unadjusted 95% interval with a Holm-adjusted p. Decide whether the interval should be adjusted too, or the check should stay on raw p.
3. Without --arm, an arm that ran only other tasks drops out of the comparison with no word (the nit at compare.py:180). Options: name the dropped arms in the header, or leave it. Recommendation: name them.
4. _cell_task assumes the arm and model slugs carry no __. A slug can produce __ from a run like _!. Not reachable with today's arm and model names; a comment in the code says so.

Test note for you: dokimi/tests on Python 3.14 hits a DeprecationWarning from langsmith (asyncio.iscoroutinefunction) at collection once litellm is installed. It is not from this PR. The compare tests pass with that warning filtered.

[antonwolfy]

Review round in eb5fbce..68b4c86, on top of the fixes in 0c7d61f..525e9d2. It settles the questions those left open:

• mde is null where the sign-flip p cannot reach 0.05 (2 / 2^k ≥ 0.05, so 5 or fewer changed tasks); the note then says how low p can go.
• A gate or score row is checked against p_holm, and its note quotes it; descriptive rows keep the raw p.
• A missing cell's task comes from its meta.json. A cell only plan.json names gets the longest known task id, so __ in a task, arm or model name no longer misplaces it.
• Without --arm, an arm with no finished cell among the --task selection is named (task_filter.dropped_arms and a header line).
• eb5fbce moves the --task selection into repos/compare_tasks.py, so compare.py stays under 800 lines.

Every thread is answered and resolved; the Testing section has the mutants and make ci-local.

[vlad-perevezentsev]

LGTM

I checked fda7c07. Everything from my earlier rounds still holds. The Holm gap from the last round is fixed: a gate or score row whose p_holm cannot drop below 0.05, even with its own p at its floor, now has no mde and a note that names Holm. mde is labelled an uncorrected t-based planning figure in the note, the contract and the changelog.

What I ran, at fda7c07:

• make ci-local --offline: 19 checks passed, 0 failed, 13 skipped (network, docker and the other Pythons). This includes the CLI snapshot and the file length report (compare.py at 800 lines).
• The two-treatment score case from the exai-bot thread: both rows now print with Holm over 2 rows, p_holm cannot go below 0.0625 here, and mde is null.
• My earlier repros again: small gate sets give mde null, the p_holm disagreement note still fires, and --task still keeps the right missing cells.
• The new test_repo_compare_reading.py on 68b4c86: the three Holm-floor and wording tests fail there and pass here.

Checked, no issues

• Holm floor: holm_floor runs correct_family with only this row's p moved to its floor. So a row that would be ranked later in Holm's order can still reach 0.05 next to a row with a very small p, and keeps its mde. A family of one row falls back to the raw rule.
• Merge from main: nothing on main touched compare.py, and the CLI snapshot now has --task.
• Docs and changelog: the contract, the module docstring, architecture.md (the new compare_reach.py) and the fragment match the code. The new file has the SPDX header.