[napetrov]

Part of #4.

Summary

Four copies of the judged-reply loop (parse, redraw, replay, reprompt) become one path:

judge._judge_answer ─┐
panel._call_judge ───┤
pairwise._both_orders┼─► judged.judged_reply(ask, prompt, parse, pinned=…)
checklist.one ───────┘        ├─ caching.first_replayed     (replay a stored redraw)
                              ├─ reprompt if pinned         (REDRAW_REMINDER)
                              └─ llm.ask_parsed             (the redraw loop)

• Judge and panel: same prompts, draws and cache keys as before; scores unchanged.
• Pairwise: at pinned sampling a redraw of the same prompt now carries a reminder, so it no longer returns the same non-answer. This is opt-in through PairwiseJudge(pinned=True).
• Pairwise and checklist reruns replay stored redraws instead of asking draw 0 live again.
• judged.py sits beside llm.py, which is over its growth ratchet.

Evidence

• Before: test_pairwise_pinned_redraws_with_a_reminder failed on main: the second prompt equalled the first (no reminder), so the redraw repeated the same reply.
• After: the same test passes; test_judge_* and test_panel_* in tests/test_judged_reply_sequence.py pin the judge and panel call sequences (prompts, draw keys, cache files, rerun call counts) and pass on both base and head.
• Full suite, two pytest seeds (--randomly-seed 11 and 22): 10937 passed, 5 skipped, 1 failed in each run: tests/test_process.py::test_sighup_kills_the_live_groups_too timed out at 30 s under full-suite load. It passes in isolation under both seeds. No file it exercises changed.
• mypy (strict, package): no issues in 286 files. ruff check on the changed files: passed. scripts/check_growth.py: the lowered baseline drops the two judge/panel entries that fell under the cap.
• CI: see the check runs on this PR.

Merge Danger

Door: two-way. No persisted format or schema change; the seam is internal.

Blast Radius: judge

• Judge and panel: no score or cache-key change for unchanged inputs.
• Pairwise: a pinned judge gets a reminder on redraw. The eval CLI does not pass pinned yet, so CLI pairwise runs are unchanged.
• Pairwise and checklist reruns make fewer requests (stored redraws replay); stored replies are not altered.
• Each unparseable pairwise draw now logs a warning from dokimi.llm; HEAD logged nothing for pairwise.
• Pairwise log wording: Pairwise order N replaces LLM in the unparseable warning.

Follow-ups (not in this PR):

• commands/judge_reliability.py: pass pinned= to PairwiseJudge when the temperature or seed is set, and pass the same to the checklist, which is still unpinned and repeats its reply at a pinned temperature.
• validity_scoring.py has its own retry loop; it is not on this path and re-asks identical replies at pinned sampling.

Checklist

• Tests added/updated and pytest passes (one load-sensitive timeout noted above)
• ruff, mypy, and schema checks pass
• Docs updated (or N/A): N/A, the changelog fragment is added
• Self-reviewed per "Before opening a PR" in AGENTS.md (two fresh-context review passes; the last one reported no blocking: findings; later commits folded its nits)
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

🤖 Generated with Claude Code

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 13 selected item(s).

Review details

• Commit reviewed: dbbcb10 against merge base 48097ae
• Coverage: 13 files in 3 groups
• Engine: OpenCodeReview v1.13.3, us.openai.gpt-6.1-sol
• Usage: 199k tokens (108k from cache), 31s
• Tool calls: 24 (17 file_read, 6 code_search, 1 file_read_diff)
• Shared judged-reply retry refactor and evaluator integration: 8 files
• Judge and grader regression test updates: 3 files
• Architecture checks and growth baseline adjustments: 2 files

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.05%. Comparing base (48097ae) to head (dbbcb10).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #314      +/-   ##
==========================================
- Coverage   94.05%   94.05%   -0.01%     
==========================================
  Files         296      297       +1     
  Lines       38026    38018       -8     
  Branches     6430     6424       -6     
==========================================
- Hits        35766    35756      -10     
- Misses       1451     1452       +1     
- Partials      809      810       +1     

Files with missing lines	Coverage Δ
dokimi/dokimi/eval/checklist.py	94.08% <100.00%> (+0.18%)	⬆️
dokimi/dokimi/eval/judge.py	94.21% <100.00%> (-0.36%)	⬇️
dokimi/dokimi/eval/pairwise.py	100.00% <100.00%> (ø)
dokimi/dokimi/eval/panel.py	97.47% <100.00%> (-0.18%)	⬇️
dokimi/dokimi/eval/rubric.py	100.00% <100.00%> (ø)
dokimi/dokimi/judged.py	100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[xaleryb]

• Ran: the full test suite, ruff, mypy and bandit at the head, plus the same on current main merged with the head; a judge/panel sequence differential against main's source; a live redraw-reminder check in both pairwise orders.
• No open must-fix or mid findings. The judge and panel prompts, draw numbers, cache keys and rerun call counts match main. When order 1 is never parseable, pairwise now skips the order-2 ask. The UnparseableVerdict is the same and it saves one call.
• No commits from us.
• Left for the author: the head conflicts with current main in dokimi/eval/judge.py, dokimi/eval/panel.py and tests/test_import_architecture.py. The conflicts come from the llm_cli/llm_json imports and the layer entries added by feat(repo): Codex and Copilot CLI harness profiles, native review mode #116/feat(llm): claude-cli provider: run the Q&A track on a logged-in Claude Code CLI #255. Keeping both sides resolves them, and the suite passes on the result.

[xaleryb]

Merged current main (63c9cdb) into this branch. Conflicts were imports only:

• eval/judge.py: kept judged_reply; added main's llm_cli import (CLI_BACKENDS, check_sampling).
• eval/panel.py: kept judged_reply; added main's check_sampling.
• tests/test_import_architecture.py: dokimi.judged added next to main's llm_cli, llm_json, process_io.

On Linux the suite, ruff, mypy, bandit and the strict benchmark run all pass.

[xaleryb]

Approving a03fd9b: CI green after main was merged in, and the automated review of this head raised nothing. No open findings from our review.