Skip to content

fix(evals): derive judge model levels from the registry (claude-opus-5 was silently demoted to medium) - #1664

Open
elhoim wants to merge 1 commit into
danielmiessler:mainfrom
elhoim:fix/evals-judge-model-registry
Open

fix(evals): derive judge model levels from the registry (claude-opus-5 was silently demoted to medium)#1664
elhoim wants to merge 1 commit into
danielmiessler:mainfrom
elhoim:fix/evals-judge-model-registry

Conversation

@elhoim

@elhoim elhoim commented Jul 27, 2026

Copy link
Copy Markdown

The three model-based graders each hardcoded a judge-model → inference-level map. The lineup moved and the maps didn't, so both current top-tier IDs fell through to the ?? 'medium' default:

judge_model resolved to should be
claude-opus-5 medium high
claude-sonnet-5 medium medium (right only by accident)

The opus case is the damaging one: an eval configured with the strongest judge silently ran on Sonnet-tier inference, and nothing surfaced the demotion — the grader still returned scores, just from a weaker model than asked for.

The fix

Derive the current lineup from EFFORT_MODEL/CURRENT in models.ts rather than restating it. That file already describes itself as the single edit point on a model release, and its header cites exactly this failure (ContextAudit checking for claude-opus-4-7 long after claude-opus-4-8 shipped) as the reason it exists.

const levelMap: Record<string, InferenceLevel> = {
  ...legacyLevels,
  ...Object.fromEntries(
    Object.entries(EFFORT_MODEL).map(([level, tier]) => [CURRENT[tier], level as InferenceLevel]),
  ),
};

Superseded IDs (claude-opus-4-8, claude-opus-4-6, claude-sonnet-4-6, and the two dated 2025 IDs) stay in a small legacyLevels table so existing eval configs keep resolving. claude-fable-5 and claude-haiku-4-5-20251001 came out of the hardcoded list because they're now produced by the derivation with identical values.

A future model bump flows through automatically instead of stranding a judge here.

Verified

Resolution asserted against the shipped expression:

  • claude-opus-5high, claude-sonnet-5medium, claude-fable-5max, claude-haiku-4-5-20251001low
  • all five legacy IDs unchanged
  • unknown model and empty string still fall back to medium

All three graders transpile clean.

Companion to #1656, which bumped CURRENT.opus to claude-opus-5 — that bump is what made the stale maps observable.

…stating them

The three model-based graders each hardcoded a judge-model -> inference
level map. The lineup moved and the maps did not, so both current top-tier
IDs fell through to the default:

  claude-opus-5    -> medium   (should be high)
  claude-sonnet-5  -> medium   (right only by accident)

The opus case is the damaging one: an eval asking for the strongest judge
silently got Sonnet-tier inference, and nothing surfaced the demotion.

Derive the current lineup from EFFORT_MODEL/CURRENT in models.ts, which
already calls itself the single edit point on a model release. Superseded
IDs stay in a small legacy table so existing eval configs keep resolving.
A future model bump now flows through automatically instead of stranding
a judge here — the same drift models.ts exists to prevent, and which its
own header cites as the cautionary example.

Verified: opus-5 -> high, sonnet-5 -> medium, fable-5 -> max, haiku -> low,
the five legacy IDs unchanged, unknown and empty still default to medium.
All three graders transpile.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant