spike(rca): abstention on healthy windows — RE3 AUC 0.93 (#118/#79)

[leo-aa88]

Gen-2 model-cycle experiment (RCAEval-only, per the plan to keep OTel untouched for later validation). Tests the abstention half of the directive — the frozen OTel run (#79) abstained on 0/2 healthy windows, because raglogs today abstains only when a window has no clusters at all (summarizer.py).

Method (RCAEval-only)

Each RCAEval case has 720–900 s of pre-injection data, so a genuinely healthy window with its own healthy baseline fits entirely before the fault:

	window	baseline
incident (positive)	[inject, inject+300]	[inject−300, inject]
healthy (negative)	[inject−300, inject]	[inject−600, inject−300]

Window-anomaly score = max over services of the strongest single-modality deviation max(log-error-rate elevation, trace-rate deviation, metric mean-change).

Result — both suites measured

suite	cases	ROC-AUC	balanced-acc @ break-even	per-system (ob / ss / tt)
RE3 (code faults)	90 / 90	0.927	89.4% (explain 79/90, abstain 82/90)	0.960 / 0.907 / 0.957
RE2 (resource/network)	270 / 270	0.917	93.3% (explain 263/270, abstain 241/270)	0.875 / 0.983 / 0.890

Generalizes across all six system×suite combos (AUC 0.88–0.98). RE2 confirms the metric/trace arm carries the metric-driven faults (its threshold 3.65 is far more portable than RE3's log-arm novel-error blow-up).

Positive result — worth building. Directly targets the frozen-eval false alarms.

Caveats for productization (documented)

• The log-arm raw score isn't portable (novel-error blow-up vs baseline ≈ 0) — a productized gate needs a bounded/robust score + a held-out threshold (same discipline as Calibrate confidence scoring against measured accuracy (or stop publishing a fake probability) #83). The RE2 metric/trace arm is already bounded and more portable.
• Abstention is a window-anomaly gate orthogonal to the ranker (fault-vs-no-fault, not which-service); they compose.
• The healthy negative is a pre-injection proxy; the remaining external check is a genuinely healthy production window on a fresh corpus.

Eval delta

None — spike + docs only, no src/core change. Full write-up in docs/spike-abstention-rca.md.

[leo-aa88]

COMMENT

The spike methodology is sound and the result is promising, but the committed write-up contradicts itself (and the PR description) on whether RE2 / the metric arm has been measured — a result doc that self-contradicts on its own headline can't merge as-is. One MAJOR reconciliation, then this is an easy approve.

Methodology (CONFIRMED sound)

• Windows are symmetric — no asymmetry bias. Incident scores [inject, inject+W] vs baseline [inject−B, inject]; healthy scores [inject−B, inject] vs [inject−2B, inject−B]; W = B = 300. Both classes get an identical 300 s-window-vs-300 s-baseline computation, only shifted.
• No fault leakage into the negatives. The healthy window ends at inject with a strict sec < we, so no t ≥ inject sample enters a healthy window; both healthy window and baseline are fully pre-injection. Genuinely healthy negatives (RCAEval's 720–900 s pre-fault data covers 2B).
• AUC is correct (Mann–Whitney U, 0.5 on ties, P(pos>neg)), threshold-free — so the 0.927/0.917 separability numbers don't depend on the in-sample threshold.
• Threshold caveat is honest. The break-even balanced-accuracy threshold is fit in-sample; the doc explicitly says a productized gate needs a held-out threshold (#83 discipline). Good — and the balanced-accuracy figures are arithmetically consistent (RE2 0.5*(263/270+241/270)=93.3%, RE3 89.4%).
• Score shape is reasonable — logs/traces contribute max(ratio−1,0) so quiet services score 0; the novel-error-rate blow-up (1e8 vs baseline≈0) is disclosed as a portability problem.
• No src/core / model change — files are docs + the standalone script only.

Findings

MAJOR (documentation integrity) — inline on the doc. The write-up asserts RE2 is both measured and not:

• Measured: result table row RE2 | 270/270 | 0.917 (line 32), caveat "Both suites now measured … the metric arm carries RE2 at AUC 0.917" (lines 65–66), headline "RE3 AUC 0.927 / RE2 AUC 0.917" (line 74).
• Not measured: "a fault that only shows in metrics (RE2-style) can still trip it — though that arm is not yet measured" (line 53), and the PR description caveat "RE2 … separability not yet measured — the required next check."

These can't both be true. The RE2 numbers are internally arithmetically consistent (93.3% checks out), so RE2 was almost certainly actually run and the two "not yet measured" sentences (doc line 53 + the PR description) are stale leftovers from the RE3-only draft. Please confirm RE2 was run and delete the stale disclaimers (or, if the RE2 row is aspirational, remove it) — as written, the doc that's supposed to record the gating result contradicts its own headline, and a merger reading the description would believe the opposite of the doc.

MINOR — _met_change maps columns via c.rsplit("_", 1)[0] (assumes {service}_{metric}); consistent with the multimodal spike, but a metric name containing _ would misattribute the service. Fine for the RCAEval melt; noting for portability.

Eval delta

Correctly None — spike + docs, no src/core. The result (if RE2 stands) is exactly the abstention signal the frozen OTel run (0/2) motivated.

VERDICT

Sound spike, promising positive result, cleanly held out of core. Not approving only because the committed result doc is self-contradictory on whether RE2/the metric arm is measured — reconcile that (near-certainly: drop the two stale "not yet measured" lines and fix the PR description) and I'll flip to approve.

[leo-aa88]

APPROVE

LGTM — re-review at 0e58a2e. The MAJOR documentation contradiction from my previous review is resolved, and the MINOR is addressed. Nothing else changed in the logic, so the methodology verdict stands.

Resolution of prior findings (CONFIRMED)

• MAJOR (RE2 measured vs not) — fixed. The stale "though that arm is not yet measured" line is now "confirmed by the RE2 run above (AUC 0.917), where the metric/trace arms carry the separation." No residual "not yet measured" for RE2/the metric arm anywhere in the doc, and the PR description was updated too ("Result — both suites measured", RE2 row present). The doc is now internally consistent and consistent with the description. The only surviving "remaining external check" is a legitimate future caveat (a genuinely healthy production window on a fresh corpus), not a contradiction.
• MINOR (rsplit service parsing) — addressed with a comment documenting the {service}_{metric} assumption and the _-in-name limitation.

Still-valid verification (unchanged; the only script delta was a comment)

• Symmetric incident/healthy windows (300 s vs 300 s), no fault leakage into negatives (strict sec < inject, both healthy window + baseline pre-injection).
• AUC is threshold-free (Mann–Whitney U), so the 0.927/0.917 separability is independent of the in-sample break-even threshold — which remains honestly caveated as needing a held-out fit (#83 discipline). Balanced-accuracy figures re-checked arithmetically (RE2 93.3%, RE3 89.4%).
• No src/core / model change.

Eval delta

None — spike + docs. The abstention signal is exactly what the frozen OTel 0/2 result motivated, now measured on both RE3 and RE2.

VERDICT

Reconciled and coherent; the positive result (RE3 0.927 / RE2 0.917, all six system×suite combos 0.88–0.98) is cleanly recorded and held out of core. Flipping my earlier COMMENT to approve.