Reviewer re-raises findings on resolved threads; severity escalates silently; confidence miscalibration

[ilopezluna]

Summary

On docker/sandboxes#4890 the review bot re-raised the same findings across multiple review passes, which the PR author read as "the bot has no memory, it's just randomness." Reconstructing the thread history shows the behavior is deterministic — but it's driven by a signal the bot ignores (thread resolution) and the contract is invisible to authors. Filing this because the evidence from that PR is unusually clean, and because the same PR surfaced a confidence-calibration failure.

Evidence: re-flagging is deterministic, keyed on "replied to" vs "silently resolved"

All data from docker/sandboxes#4890 review threads (GraphQL reviewThreads, isResolved / isOutdated / comment authors).

Control group — findings that were answered and fixed never resurfaced. Five Copilot comments (2026-08-06) each got an author reply plus a code change. None were ever re-raised by any reviewer pass.

Findings resolved with no reply and no code change were re-derived every pass:

Finding	First raised	Author action	Re-raised
genericModelMixinTemplate missing SBX_CRED_*_MODE	08-11 23:03 UTC	resolved silently, code unchanged	08-12 09:07 (same finding), 08-12 10:55 (codex/opencode variants)
Dockerfile LLMMAN_REPO default switched to personal repo	08-12 08:25 as [high]	resolved silently, code unchanged	08-12 10:55, escalated to [HIGH/security]

The findings themselves were substantively correct — the author's final push (08-12 11:17 UTC) adopted all the SBX_CRED_*_MODE suggestions, with code comments restating the bot's rationale. So the bot was right three times and got dismissed three times, and each re-derivation cost author goodwill instead of earning credit.

Problems

1. Resolve-without-reply is not treated as a dismissal signal. Re-review of unchanged code re-derives the same finding from scratch. Authors experience this as amnesia; several are now advocating removing the bot from docker/sandboxes despite a good hit rate on this PR (6 findings, 5 adopted).
2. Silent severity escalation on re-raise ([high] → [HIGH/security] for the identical finding on unchanged code) reads as nagging, not review.
3. Confidence calibration. The Dockerfile finding claimed personal GitHub repos don't support branch/tag protection, stated with maximal confidence; that's been wrong since ~2020, and the bot apologized when challenged (docker/sandboxes#4890, Dockerfile line 310 thread). One confidently-wrong claim erases the credibility of the correct findings around it. The finding was also over-severe on its own terms: the target was pinned by full commit SHA, so branch protection was moot for integrity.

Suggestions

• Treat a human resolving a bot thread as an explicit dismissal: suppress re-derived duplicates of resolved findings when the flagged lines are unchanged (or downgrade them to a one-line "previously raised and resolved without changes" note).
• Never silently escalate severity of an unchanged, previously-dismissed finding.
• When a finding materially depends on a checkable fact (GitHub feature availability, pinning semantics), verify or hedge instead of asserting at 100/100.
• Document the memory model somewhere authors can see it (e.g. in the review comment footer): what the bot remembers across pushes, and what resolution does and doesn't do.