fix(evals): derive judge model levels from the registry (claude-opus-5 was silently demoted to medium) - #1664
Open
elhoim wants to merge 1 commit into
Open
Conversation
…stating them The three model-based graders each hardcoded a judge-model -> inference level map. The lineup moved and the maps did not, so both current top-tier IDs fell through to the default: claude-opus-5 -> medium (should be high) claude-sonnet-5 -> medium (right only by accident) The opus case is the damaging one: an eval asking for the strongest judge silently got Sonnet-tier inference, and nothing surfaced the demotion. Derive the current lineup from EFFORT_MODEL/CURRENT in models.ts, which already calls itself the single edit point on a model release. Superseded IDs stay in a small legacy table so existing eval configs keep resolving. A future model bump now flows through automatically instead of stranding a judge here — the same drift models.ts exists to prevent, and which its own header cites as the cautionary example. Verified: opus-5 -> high, sonnet-5 -> medium, fable-5 -> max, haiku -> low, the five legacy IDs unchanged, unknown and empty still default to medium. All three graders transpile.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The three model-based graders each hardcoded a judge-model → inference-level map. The lineup moved and the maps didn't, so both current top-tier IDs fell through to the
?? 'medium'default:claude-opus-5mediumhighclaude-sonnet-5mediummedium(right only by accident)The opus case is the damaging one: an eval configured with the strongest judge silently ran on Sonnet-tier inference, and nothing surfaced the demotion — the grader still returned scores, just from a weaker model than asked for.
The fix
Derive the current lineup from
EFFORT_MODEL/CURRENTinmodels.tsrather than restating it. That file already describes itself as the single edit point on a model release, and its header cites exactly this failure (ContextAuditchecking forclaude-opus-4-7long afterclaude-opus-4-8shipped) as the reason it exists.Superseded IDs (
claude-opus-4-8,claude-opus-4-6,claude-sonnet-4-6, and the two dated 2025 IDs) stay in a smalllegacyLevelstable so existing eval configs keep resolving.claude-fable-5andclaude-haiku-4-5-20251001came out of the hardcoded list because they're now produced by the derivation with identical values.A future model bump flows through automatically instead of stranding a judge here.
Verified
Resolution asserted against the shipped expression:
claude-opus-5→high,claude-sonnet-5→medium,claude-fable-5→max,claude-haiku-4-5-20251001→lowmediumAll three graders transpile clean.
Companion to #1656, which bumped
CURRENT.opustoclaude-opus-5— that bump is what made the stale maps observable.