feat: replace GPT-5.4-mini experiments with GPT-5.6 Luna - #269
Open
Rodriguespn wants to merge 6 commits into
Open
feat: replace GPT-5.4-mini experiments with GPT-5.6 Luna#269Rodriguespn wants to merge 6 commits into
Rodriguespn wants to merge 6 commits into
Conversation
Renames the four 5.4-mini experiments to gpt-5.6-luna per the new Assistant default (supabase/supabase#49749), matching the Assistant reasoningEffort (medium). Drops the stale codex-gpt-5.4-mini results from eval-results.json; CI (run-evals label) will regenerate real results for the new experiments.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
mattrossman
approved these changes
Sep 7, 2026
mattrossman
reviewed
Sep 7, 2026
Clarifies naming alongside the new codex-gpt-5.6-luna experiments, per review feedback on #269. Pure rename: model, suite, skills, and runtime config are unchanged.
Removes experiments never wired into any experiment suite (benchmark, no-skills, regression) and therefore never run by the automated eval refresh: claude-haiku-4.5, claude-opus-4.7, claude-sonnet-5, claude-sonnet-5-executor, openai-gpt-5.4-nano, openai-gpt-5.6, openai-gpt-5.6-luna, openai-gpt-5.6-luna-executor. None have historical results in eval-results.json / regression-eval-results.json and none are referenced in docs. This drops the openai-gpt-5.6-luna variants added earlier in this PR; the codex-gpt-5.6-luna / codex-gpt-5.6-luna-no-skills experiments (wired into benchmark/no-skills) remain as the covered gpt-5.6-luna coverage for AI-1190.
…luna # Conflicts: # apps/web/src/data/regression-eval-results.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the four GPT-5.4-mini experiments with GPT-5.6 Luna, the new Assistant default model, so the model change in supabase/supabase#49749 is covered by the public eval suite.
Why
The Assistant default model changed to
gpt-5.6-luna(supabase/supabase#49749); the public evals suite should track that model.Closes AI-1190