FEAT: Scorer Observation (Phase 3) - #2630
Merged
Richard Lundeen (richlundeen) merged 8 commits intoSep 15, 2026
Merged
Richard Lundeen (richlundeen) merged 8 commits into
Richard Lundeen (richlundeen) merged 8 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8dd850ac-5e7b-4c35-bdb4-a0a9e811d020
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8dd850ac-5e7b-4c35-bdb4-a0a9e811d020
Roman Lutz (romanlutz)
left a comment
Roman Lutz (romanlutz)
left a comment
Contributor
There was a problem hiding this comment.
A few comments of my own and a few from GHCP that I agree with 🙂
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8dd850ac-5e7b-4c35-bdb4-a0a9e811d020
Make observation persistence and retention transactional, require explicit subclass replay contracts, and use target-independent judgment names. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8dd850ac-5e7b-4c35-bdb4-a0a9e811d020
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8dd850ac-5e7b-4c35-bdb4-a0a9e811d020
Roman Lutz (romanlutz)
approved these changes
Sep 12, 2026
Clarify expectation-bound replay and cover rescoring stored evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8dd850ac-5e7b-4c35-bdb4-a0a9e811d020
Place the unpublished observation migration after the upstream attack score split, preserving one Alembic head. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8dd850ac-5e7b-4c35-bdb4-a0a9e811d020
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds durable LLM scorer observations, atomic score-to-observation persistence, and replay of stored judgments without another model call. It preserves exact scored evidence, scorer configuration, expectation, and response-parser contracts as described in the original Phase 3 design gist. This PR is intentionally large because the observation model, database migration, memory invariants, scorer integrations, documentation, and focused regression tests must ship together as one usable feature.