Skip to content

fix(evals): avoid overriding judge reasoning - #6566

Merged
tinalenguyen merged 2 commits into
livekit:mainfrom
inickt:fix-judge-responses-options
Jul 29, 2026
Merged

fix(evals): avoid overriding judge reasoning#6566
tinalenguyen merged 2 commits into
livekit:mainfrom
inickt:fix-judge-responses-options

Conversation

@inickt

@inickt inickt commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Stop the generic eval judge from injecting the Chat Completions-specific reasoning_effort="none" option.
  • Preserve reasoning options configured on the supplied LLM.
  • Add regression coverage for the options forwarded by the judge.

Why

The judge accepts any LLM, but reasoning_effort is specific to Chat Completions. Passing it to an OpenAI Responses client fails because Responses uses the reasoning object instead.

Explicitly configured reasoning options continue to apply. LLMs without an explicit configuration will now use their provider or model default rather than having the judge force reasoning off.

Testing

  • Added focused unit coverage.
  • Ran the full formatting, linting, and type-check suite.

@inickt
inickt marked this pull request as ready for review July 27, 2026 14:54
@inickt
inickt requested a review from a team as a code owner July 27, 2026 14:54

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

@tinalenguyen tinalenguyen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you for the PR, this change makes sense. i think maybe we should set the default reasoning for inference GPT 5 models to none before applying this change so the judges don't suddenly start with medium effort. i can make that PR

@tinalenguyen
tinalenguyen merged commit 0390cdb into livekit:main Jul 29, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants