Skip to content

Describe video frames with OpenAI reasoning models - #456

Merged
ArnavAgrawal03 merged 2 commits into
mainfrom
arnav/frame-reasoning-models
Oct 26, 2026
Merged

ArnavAgrawal03 merged 2 commits into
mainfrom
arnav/frame-reasoning-models

Conversation

@ArnavAgrawal03

@ArnavAgrawal03 ArnavAgrawal03 commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Video frame descriptions failed on OpenAI reasoning models. VisionModelClient.get_frame_description always sent max_tokens = 300, and gpt-5 and later reject it:

litellm.BadRequestError: OpenAIException - Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.

Image OCR already handled this (#454). This moves that logic into core/parser/model_params.py and uses it for both.

Changes

  • add_output_limit sets max_completion_tokens for OpenAI models and max_tokens for other providers, keeps a limit set on the registered model, and allows reasoning_effort through LiteLLM. Image OCR calls it instead of its inline copy, so OCR behavior is unchanged.
  • Frame descriptions get a 4096-token budget instead of 300, so a reasoning model has room to think before it writes. Ollama models still get no limit, as before.
  • A description that runs out of output with no text raises a ValueError that suggests reasoning_effort = "low", instead of indexing a blank frame.
  • Rate limits, timeouts and outages raise ConnectionError, so the ingestion worker retries, as for OCR.
  • DOCKER.md says reasoning models work for frames and recommends reasoning_effort = "low".

Testing

  • test_video_transcription.py adds 4 tests: OpenAI reasoning model parameters, other providers and registered limits, running out of output, and transient errors. With test_video_parser and test_image_ocr_model: 37 passed.
  • Live calls through get_frame_description on a generated frame with a red "Error 4012: Invoice total does not match" box. openai/gpt-6.1-sol returned the error above before this change. After it, openai/gpt-6.1-sol (with and without reasoning_effort = "low") and gpt-4.1-mini all described the box and quoted its text.
  • ruff with its long-standing default rules (E4, E7, E9, F) passes on the changed files.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

ArnavAgrawal03 and others added 2 commits October 26, 2026 10:11
The frame description call always sent max_tokens = 300, which gpt-5 and later OpenAI models reject. It now
sets the output limit the way image OCR does: max_completion_tokens for OpenAI, max_tokens elsewhere, allowing
reasoning_effort through LiteLLM. The shared logic moves to core/parser/model_params.py. The budget is 4096 so a
reasoning model has room to think; a description that runs out of output fails instead of indexing blank text,
and rate limits and outages are retried.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ArnavAgrawal03
ArnavAgrawal03 merged commit 3b048d0 into main Oct 26, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant