[Improve] Ground PR review pre-screen hints in specific hunks

[mrubens]

What changed

Follow-up to #3119. The pre-screen now points the reviewer at specific changed hunks instead of whole-PR risk categories. The selection rule was tuned and checked against past review findings.

• Hunk-level hints. The diff is split into per-file hunks. For each hunk the decision model answers two questions: are there concrete defects in these changed lines, and if so, what kind. Style, naming, missing tests and "sensitive area" count as no. The reviewer gets the top 3 hunks (at most 2 per file), each with file, head-side line range, hunk header, rank among the screened hunks, and the likely defect area when the model is confident about it. There is no "looks safe" output, and the reviewer is told that unflagged code gets the same depth of review.
• Rank, not an absolute cutoff. The model's defect probabilities are compressed: almost no hunk reaches 0.7, so [Feat] Add decision-model pre-screen to pull-request reviews #3119-style thresholds produced 3 hints in 120 PRs. Ranking within a PR is strong, so hints are the top-ranked hunks, with a 0.1 floor that only drops hunks the model is confident are clean. Area confidence did not predict hint accuracy (about 19% vs 20% precision above and below 0.5), so it only decides whether the area label is shown.
• Oversized hunks are chunked at 150 lines. Chunk boundaries never split a removed/added replacement. Mostly this affects whole new files, so a hint points at a region rather than an entire file.
• Fair diff sampling. The first-20k-characters cap is replaced. Hunks are picked round-robin across files (up to 64), and a 48k-character budget is split max-min fairly, so one large file cannot starve later files. Lockfiles and generated assets are skipped. Git-quoted paths (spaces, non-ASCII) are decoded. Deletion-only hunks keep a zero-length range and are described as "lines removed after line N".
• Batching. Requests go in parallel batches of 32 hunks. The timeout is raised to 10s.
• Main-review effort is unchanged. Hints do not select a model, reasoning effort, or scope.
• Eval harness. pnpm --filter @roomote/cloud-agents review-prescreen:eval --repo owner/name --reviewer login[,login] [--limit N] [--chunk-lines N] [--out scores.json]. It replays merged PRs at the commit their first review ran against, runs the shipped selection code, and scores hints against that review's inline comments. Those reviews ran without hunk hints, so the ground truth is independent of the pre-screen.

Example output

This is the review_prescreen text the reviewer receives, generated with this branch for past PRs, followed by what the first review actually flagged on the same commit.

#3119 (28 screened hunks). The review's one finding, at line 245 of the new file, is inside hint 3. Chunking narrowed that hint from the whole 256-line file to lines 151-256.

Advisory decision-model pre-screen (untrusted triage, not review findings). These changed hunks ranked most likely to contain a defect; inspect them early:
- `packages/cloud-agents/src/server/workflows/githubPrReview.ts` lines 432-445 (`@@ -431,8 +432,14 @@ export async function githubPrReview({`): ranked 1 of 28 screened hunks.
- `packages/cloud-agents/src/server/workflows/githubPrReviewPrescreen.ts` lines 1-150 (`@@ -0,0 +1,150 @@`): ranked 2 of 28 screened hunks, most likely a correctness issue.
- `packages/cloud-agents/src/server/workflows/githubPrReviewPrescreen.ts` lines 151-256 (`@@ -0,0 +151,106 @@`): ranked 3 of 28 screened hunks, most likely a correctness issue.
Treat each line as a question to verify, not a finding: report it only if the code confirms a concrete defect. The pre-screen misses about half of real findings and never clears code, so give unflagged hunks and files the same depth of review.

#3066 (9 screened hunks). Both findings are hinted: a retry storm after a failed cache refresh (line 66, hint 1, labeled concurrency) and a process-wide cache retaining credentials (line 654, hint 2).

- `packages/cloud-agents/src/server/fast-agent/fast-agent-prompt-skill-snapshot-cache.ts` lines 1-116 (`@@ -0,0 +1,116 @@`): ranked 1 of 9 screened hunks, most likely a concurrency or lifecycle issue.
- `packages/cloud-agents/src/server/fast-agent/fast-agent-repository-skill-source.ts` lines 649-681 (`@@ -648,3 +649,33 @@ export class RemoteFastAgentRepositorySkillSource implements FastAgentRepository`): ranked 2 of 9 screened hunks.
- `packages/cloud-agents/src/server/fast-agent/fast-agent-settings-skill-source.ts` lines 778-805 (`@@ -777,3 +778,28 @@ export class RemoteFastAgentSettingsSkillSource implements FastAgentSettingsSkil`): ranked 3 of 9 screened hunks.

#3071 (35 screened hunks). This is a partial miss. Hint 2 covers the follow-up dedupe bug at harness line 5393. The unredacted diff_stat at task-completion-gate.ts:184 falls just outside hint 3 (lines 1-150), and a stale test assertion in a workflow doc was not hinted.

- `apps/worker/src/sandbox-server/lib/harnesses/opencode-server/harness.ts` lines 5487-5510 (`@@ -5364,18 +5487,24 @@ export class OpenCodeServerHarness`): ranked 1 of 35 screened hunks, most likely a correctness issue.
- `apps/worker/src/sandbox-server/lib/harnesses/opencode-server/harness.ts` lines 5364-5454 (`@@ -5326,6 +5364,91 @@ export class OpenCodeServerHarness`): ranked 2 of 35 screened hunks.
- `packages/cloud-agents/src/server/task-completion-gate.ts` lines 1-150 (`@@ -0,0 +1,150 @@`): ranked 3 of 35 screened hunks, most likely a correctness issue.

Validation

Scored against the 200 most recent merged PRs in this repo that got inline review comments: about 5,700 hunks, 210 of which held findings (3.7% base rate). The top-3 rule was chosen on the 120 most recent PRs and then checked on the next 80.

	Hint precision	Finding-hunk recall	Random-3 recall	Largest-3-hunks recall	PRs where a hint hit
Tuning (120 PRs)	18.8%	50.0%	20.7%	40.9%	65/120
Holdout (80 PRs)	20.2%	61.5% (95% CI 51-72%)	25.1%	46.2%	48/80

• Precision is about 5x the base rate. Within-PR AUC has a median of 0.91 (pooled 0.78). On the tuning set, the model beats a "largest hunks" baseline on recall@3 by 11 points (bootstrap 95% CI +4 to +20).

Latency. Wall time for all decision-model requests for one PR, measured against TypeSafe's API over the 200 eval PRs (150-line chunking):

	p50	p90	p95	p99	Max
All PRs	322ms	651ms	754ms	1,132ms	1,161ms
Up to 32 hunks, 1 request (120 PRs)	247ms		520ms
33-64 hunks, 2 parallel requests (80 PRs)	479ms		1,008ms

The pre-screen starts right after the diff fetch and runs alongside the review-comment and issue-comment fetches, so it adds little or no wall time before the reviewer starts. The 10s timeout never came close to firing. Deployments that reach the model through OpenRouter, Vercel, or a self-hosted upstream will add their own network overhead.

Chunk size, compared on the same 197 PRs using finding-level coverage (share of the 237 inline findings inside a hinted range):

Chunking	Finding coverage	Hint precision	Median / p90 hinted lines per PR	Findings per 1k hinted lines
None	49.1%	19.1%	112 / 316	3.6
150 lines (shipped)	49.1%	19.1%	112 / 259	4.5
50 lines	41.0%	16.0%	82 / 129	5.9

The shipped row is from a rerun after the replacement-boundary fix; only 2 PRs chunked differently, and the change is within run-to-run variation (about half a point). 50-line chunks lose about 8 points of coverage (95% CI -12 to -4): small chunks strip the context the model ranks with, and within-PR AUC drops from 0.91 to 0.89. 150-line chunks keep coverage (difference CI -3.4 to +2.6 points) and trim the largest hinted regions.

Unit tests cover parsing, quoted paths, deletion-only ranges, chunking and chunk boundaries, round-robin selection, fair budgeting, ranking, caps, area-label withholding, batching, and fallback.

Limits

• Ground truth is limited to PRs where the reviewer left at least one inline comment, so this does not measure whether hints cause false positives on clean PRs.
• It measures whether hints point at the right code, not whether the reviewer finds more with them. The next step is paired end-to-end reviewer runs on frozen PRs, with and without hints, before any use of hints to lower review effort.

[roomote-community]

No new code issues found. See task

• Quoted file paths are silently excluded from hunk screening.
• Deletion-only hunks receive invalid head-side line ranges.
• Deletion-only hunks are described as occurring after the wrong head-side line. — dismissed: Git numbers a zero-length head range from the preceding retained line, so the existing wording is correct.
• Large removed/added replacements can still be split between chunks after the 10-line boundary overrun.

_{Reviewed f103b41}