fix(multiscan): record run warnings in the campaign ledger - #295
Conversation
Fixes openai#248. `runMultiscan` called `security.run()` with no `onWarning` observer, so every warning the scan raised was dropped. A repository whose target drifted mid-run still completes, so it landed in results.jsonl as `status: "completed"` with nothing anywhere in the campaign output saying the results describe a tree that moved. bulk-scan is the mode where nobody is watching an individual repository, which is what makes the silence expensive. The receipt now carries a `warnings` array when the scan raised any, each entry `{ message, kind? }`. `kind` is preserved from ScanWarningDetails so a consumer can single out drift without matching on message text, which is what the CLI already does at the other call site. Three details worth review: - Warnings go through `redactedErrorMessage`, the same redaction the failure path already applies. The ledger is a file on disk, and a warning can quote a remote or a token, so it gets the same treatment as an error. The CLI runs its warnings through `sanitizeDiagnosticValue`, but that is private to cli.ts; `redactedErrorMessage` is the equivalent already imported here. - The array is per attempt, declared beside `failure` and `cost`, so a retry does not inherit the previous attempt's warnings. - `status` is untouched. The scan did complete, and turning a warning into a failure would be a different and much louder change than the issue asks for. The key is omitted entirely when nothing was raised, so existing consumers see no new field on a quiet run. Three tests added. Two of them fail against the current code, which is the point; the third is the quiet-run control that passes either way and guards against warnings appearing spuriously. Verified on Windows 10, bun 1.3.11: 16 pass with the fix, 14 pass and the two new warning tests fail without it. The single unrelated failure in both runs is "rejects output-directory symlinks", which needs symlink privileges this host does not grant and fails identically on unmodified main.
main added a coverage warning path since this branch opened: `warning?: string` on the receipt, a `completed_with_incomplete_coverage` status, and an `incomplete` tally. Three conflicts, all in multiscan.ts. Resolved by keeping both signals separate rather than folding one into the other, because their semantics differ in a way that matters: `warning` is derived locally from `result.coverage.completeness`, and its presence flips `status` to `completed_with_incomplete_coverage`. The scan's own warnings must not do that. A repository whose target drifted mid-run has *complete* coverage — the scan reviewed everything it set out to review, the tree moved underneath it — so routing a drift warning into `warning` would report incomplete coverage for a run whose coverage was fine. So the observer-sourced warnings land in `scanWarnings`, renamed from `warnings` to remove the collision with main's singular `warning`. The name now says which layer raised it: the campaign derived `warning`, the scan raised `scanWarnings`. main's `scanPrompt` / `postScanPrompt` options are kept as-is; the `onWarning` observer is added alongside them. The underlying defect from openai#248 is untouched by main's work: `security.run()` still receives no `onWarning`, so warnings the scan raises are still discarded. Coverage was a locally computed signal and never went through that path. 38 pass. The single failure is `rejects output-directory symlinks`, which needs symlink privileges this host does not grant and fails identically on unmodified main.
6cfdf7b to
6fdb94d
Compare
|
Rebuilt against
It does not cover what this PR is about. I kept the two signals separate rather than folding mine into
So the observer-sourced entries land in
38 pass. The one failure is Happy to merge the two into a single field if you would rather have one, but it would need the status coupling loosened first, and that felt like a larger decision than this PR should make on its own. |
|
Closing in favour of #255, which was open before this and covers the same ground. I did not check for PRs already linked to the issue before opening this, which I should have. Apologies for the extra review load. |
Fixes #248.
runMultiscancalledsecurity.run()with noonWarningobserver, so every warning the scan raised was discarded. A repository whose target drifted mid-run still completes, so it landed inresults.jsonlasstatus: "completed"with nothing in the campaign output recording that the results describe a tree that moved.The receipt now carries a
warningsarray when the scan raised any:{"id":"drifted","status":"completed","attempt":1,"outputDir":"...", "warnings":[{"message":"Scan target changed during the run.","kind":"target_changed"}]}kindcomes fromScanWarningDetails, so a consumer can single out drift without matching on message text — the same distinction the CLI already makes at its ownonWarning.Three decisions worth review
Redaction. Warnings go through
redactedErrorMessage, the redaction the failure path already applies. The ledger is a file on disk and a warning can quote a remote or a token. The CLI usessanitizeDiagnosticValuefor this, but that is private tocli.ts;redactedErrorMessageis the equivalent already imported intomultiscan.ts. Happy to export the CLI one instead if you would rather have a single sanitizer.Per attempt. The array is declared beside
failureandcostinside the retry loop, so a second attempt does not inherit the first attempt's warnings.statusis untouched. The scan completed; promoting a warning to a failure would be a louder change than the issue asks for and would break resume, sincecompletedreceipts are what let a rerun skip finished repositories. The key is omitted entirely when nothing was raised, so a quiet run produces byte-identical output to today.Verification
Windows 10, bun 1.3.11,
bun test tests-ts/multiscan.test.ts:warningswhen none raisedThe two failing-without-the-fix tests are the ones that matter; the third is a control that passes either way and guards against warnings appearing on a quiet run.
The single unrelated failure in both columns is
rejects output-directory symlinks, which needs symlink privileges this host does not grant. It fails identically on unmodifiedmain, so it is not residue from this change.Related
This is the multiscan half of the same shape as #251 and #195: the tool knew something and the knowledge did not reach the artifact a consumer reads. Those are about the single-scan
--jsonand SARIF surfaces; this one is the ledger. I have not touched either of those paths — #251 already has someone working on it.