Repository navigation
GeminiAgent: Gemini 4 request contract, streaming, and long-decode continuation - #724
jdchawla29 wants to merge 4 commits into
Conversation
…nued cap as length GeminiConfig no longer sends temperature 1.0, top_p 0.95, and top_k 40 on every request; unset, each model uses its own sampling. Gemini 4 models ignore sampling parameters, and Gemini 3's recommended temperature is its default. A Gemini 4 turn that reaches the per-request output cap ends with finish reason CONTINUATION. When the client cannot continue it (a google-genai older than 2.30.1), the turn is a truncation, so the run now reports stop_reason=length instead of an empty "done".
GeminiAgent made one non-streaming generateContent call per turn. A Gemini 4 turn of 32,768 output tokens took about 645 seconds in measurement, longer than non-streaming request timeouts in front of the model (the HUD gateway allows 600 seconds), and Google recommends streaming for long generations. Each turn now streams and joins the streamed parts into one model turn, kept as received so thought signatures stay on the parts they arrived with. Text and thought fragments concatenate without separators, since streamed fragments split mid-word. Usage, grounding, and finish reason come from the last chunk that carries them.
anthropic 1.x renamed the request TypedDict for citations to
BetaCitationsConfigParamParam; BetaCitationsConfigParam is now the response
model. A plain {"enabled": True} is the citations parameter on both 0.x and 1.x.
Every browser-use release pins google-genai below 2.0 (0.11.13) or to 1.65.0 (0.12.1 and later), and the dev environment resolved together with the browseruse extra. That held SDK tests at google-genai 1.75, openai 2.x, and anthropic 0.x, while installing hud without that extra gets the latest of each. The browseruse extra now resolves apart from dev, and CI runs every other extra. dev requires google-genai 2.30.1 or later, so GeminiAgent's tests cover a stream that stops at the per-request cap and continues with its continuation token as one turn. hud's own google-genai floor is unchanged, so hud[browseruse] still installs. On openai 3.x and anthropic 1.x the SDK clients run on httpx2, so the tests that inject transports or build error responses use httpx2.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6a0bfa819f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # GeminiAgent's long-decode tests need the SDK's automatic continuation. | ||
| "google-genai>=2.30.1", |
There was a problem hiding this comment.
Adding google-genai>=2.30.1 only to the dev extra leaves normal installs—including hud[browseruse] and environments already locked to the declared core floor of 1.75—on SDK versions that predate FinishReason.CONTINUATION. When Gemini 4 reaches its per-request cap, those versions fail while validating the streamed response instead of yielding a chunk that can be mapped to stop_reason="length", so the rollout is recorded as an error and the documented fallback never runs. Either the shipped dependency contract must require a continuation-aware SDK or the response path must explicitly support older versions.
AGENTS.md reference: AGENTS.md:L7-L9
Useful? React with 👍 / 👎.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6a0bfa8. Configure here.
| top_k: int = 40 | ||
| temperature: float | None = None | ||
| top_p: float | None = None | ||
| top_k: int | None = None |
There was a problem hiding this comment.
Long decode stays off by default
High Severity
GeminiConfig.max_output_tokens stays unset, and that value is forwarded on every GenerateContentConfig. Gemini 4 long decode only starts when max_output_tokens is above 65536; below that the API stays on the 64K per-request cap and will not emit a continuable hop. Default Gemini 4 turns therefore stop at that cap even with google-genai 2.30.1+, which is the opposite of the documented automatic continuation.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 6a0bfa8. Configure here.
This branch has not been deployed
1 participant
SDK half of HUD-3143 (Migrate the Gemini integration to Google's current GenAI platform) and HUD-3144 (Support Gemini 4 long decode). Gateway side: hud-evals/hud-monorepo#2308.
Summary
GeminiConfig.temperature,top_p, andtop_kdefault to unset. GeminiAgent stopped sending 1.0 / 0.95 / 40 on every request; Gemini 4 ignores sampling parameters, and Gemini 3's recommended temperature is its default anyway.thinking_levelkeeps the union of levels. Gemini returns a clear 400 for a level a model lacks (Argon rejectsminimal).CONTINUATION. With google-genai 2.30.1 or later, the SDK resumes with the continuation token and GeminiAgent records the hops as one turn. If the installed SDK can't continue, the turn is reported asstop_reason="length"instead of an emptydone.browseruseextra. That held SDK tests at google-genai 1.75, openai 2.x and anthropic 0.x, while installing hud without that extra gets the latest of each. Thebrowseruseextra now resolves apart fromdev([tool.uv] conflicts), CI runs every other extra, anddevrequires google-genai 2.30.1 or later for the continuation test. hud's own google-genai floor is unchanged, sohud[browseruse]still installs.citationsfrom a class that anthropic 1.x turned into a response model. It's now a plain dict that is valid on both majors.Testing
Run locally against the dependency set CI will resolve: google-genai 2.31.0, openai 3.31.0, anthropic 1.14.0, everything else up to Oct 8.
uv run pytest: 1,612 passed, 6 skipped.ty check --error-on-warning,ruff format --check,ruff check: clean.CONTINUATIONand resumes with its token through the real SDK (second request is the first pluscontinuationToken; usage summed);CONTINUATIONcounted as truncation.MALFORMED_FUNCTION_CALL, buffered or streamed alike (1 in 8 vs 0 in 8 on the same request), which is model behavior.Notes
include_thoughts=False, a streamed turn sends nothing while the model thinks. A long silent think can hit a gateway's first-chunk deadline, which is 240 seconds on the HUD gateway. Documented indocs/v6/reference/agents.mdx.🤖 Generated with Claude Code
Note
Medium Risk
Changes core Gemini inference (streaming, message assembly, truncation semantics) and dependency resolution for dev/CI; rollout behavior and gateway timeouts depend on the new path.
Overview
GeminiAgent now uses streaming (
generate_content_stream) per turn so long generations stay under gateway timeouts. Streamed parts are merged into a single model message with thought signatures preserved on the parts they arrived on; text and reasoning fragments are concatenated (not newline-joined).Gemini 4 contract:
GeminiConfigleavestemperature,top_p, andtop_kunset by default instead of sending fixed sampling values. Default model in docs isgemini-3.1-pro-preview. Withgoogle-genai≥2.30.1, per-request output caps can auto-continue via the SDK; unfinishedCONTINUATIONis treated as truncation (stop_reason="length").Tooling & CI:
TRUNCATION_FINISH_REASONSincludes GeminiCONTINUATION. Claude web-fetchcitationsis a plain{"enabled": True}dict for anthropic 1.x compatibility. Tests that mock HTTP use httpx2 (openai 3.x / anthropic 1.x). CI runs pytest with--no-extra browseruse;pyproject.tomldeclares a browseruse vs dev uv conflict and pinsgoogle-genai>=2.30.1in dev for continuation tests.Docs add Gemini sampling, thinking levels, streaming, and
include_thoughtsguidance.Reviewed by Cursor Bugbot for commit 6a0bfa8. Bugbot is set up for automated code reviews on this repo. Configure here.