Repository navigation
fix(gradium): STT cumulative interims, vad_bucket=0, connect timeout, websocket hardening (2/2) - #7871
Draft
hhugo-gradium wants to merge 9 commits into
Draft
fix(gradium): STT cumulative interims, vad_bucket=0, connect timeout, websocket hardening (2/2)#7871hhugo-gradium wants to merge 9 commits into
hhugo-gradium wants to merge 9 commits into
Conversation
The STT stream looked up `delay_in_tokens` in the ready message, a key the server never sends (it is `delay_in_frames`), so the hard-coded fallback of 6 steps was always used. Production models run with a delay of 10, so a turn was finalized 4 steps (320ms) before the model had emitted its last words, which then showed up as a spurious new turn. Adds a scripted fake websocket harness for the STT tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
When the input channel closed, send_task returned without telling the
server; `closing_ws` was never set, so recv_task polled until the server
dropped the idle socket, which was then reported as an unexpected close
and retried against a closed input. The unused `_CLOSE_MSG`
(`terminate_session`) was another provider's protocol.
Send `{"type": "end_of_stream"}` once the input ends, mark the socket as
closing, and on the server's end_of_stream emit the buffered words as
the final transcript before returning. Raise APIError with the server's
message on `error` frames instead of logging them as unknown. Skip the
VAD silence flush once the input channel is closed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
They only talk to fake websockets, so they belong in the hermetic category that CI runs rather than the provider category. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Only the server's end_of_stream path emitted the words buffered since the last VAD finalization. The two other clean exits after end_input(), a close frame instead of the reply and the receive timeout, dropped them without a FINAL_TRANSCRIPT. Finalize on all three. A server-initiated end_of_stream while audio is still being sent (a session duration limit, for instance) was treated as a clean end: the receive task returned and the next audio write failed with a raw connection error the base class does not retry. Finalize, then raise a retryable APIStatusError so the stream reconnects. The final end_of_stream send is guarded: if the server closed the socket at the same moment, that is not an error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Each `text` piece from the server was emitted as its own INTERIM_TRANSCRIPT. The agent treats an interim as the whole partial utterance: it replaces the previous one for live captions and, when the final transcript is late, becomes the user's turn text. One-word interims made captions flicker and could hand the LLM only the last word. Emit the joined buffer of the current utterance, with the utterance's start time, on both interim and final events. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`vad_bucket` was tested for truthiness, so selecting bucket 0 (the 0.5s horizon) silently disabled end-of-turn detection and no FINAL_TRANSCRIPT was ever emitted. Only None means disabled. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ws_connect ran under the shared aiohttp session's default timeout of five minutes, so a hung handshake blocked the stream for minutes and conn_options.timeout was never applied. Wrap the connect in asyncio.wait_for and map timeouts and handshake failures (401, 402, ...) to APITimeoutError and APIStatusError so the base class retry rules apply instead of a raw aiohttp exception. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Bring the STT stream in line with the other websocket STT plugins: - Map every connect failure, not only timeouts and HTTP rejections, to APIConnectionError: a DNS failure, a refused connection or a TLS error raised an aiohttp exception the base class treats as fatal on the first attempt. - Open the socket with a 30s heartbeat. Without it a silently dropped socket was never noticed: the receive loop polled forever and audio kept going into a dead socket, with no error and no reconnect. - Handle the ERROR frame the heartbeat produces, instead of logging it as an unexpected type and losing the reason. - Guard the audio writes: a mid-write socket drop raised a raw connection error; it is now a retryable APIConnectionError, or a quiet return when the stream was closing anyway. - Read a text frame's fields before flipping the speaking state, so a malformed frame cannot leave the turn started without a START_OF_SPEECH. - Drop the unused `_speech_duration` and `bytes_per_frame`, align the STTOptions vad_threshold default with the constructor's, and fix the stale "only final transcripts" comment. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`update_options(buffer_size_seconds=...)` set the reconnect event, so the stream tore down its websocket and opened a new one. The receive task was cancelled without finalizing, which dropped the words buffered for the utterance in progress: live, an update mid-sentence lost the first half of the transcript (WER 0.58 against 0.11 for the same audio). Nothing `update_options` changes is used by the live session (`buffer_size_seconds` is only stored), so there is nothing to reconnect for. Keep the socket, and drop the reconnect loop that only this path drove; the base class already retries a failed session. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
hhugo-gradium
force-pushed
the
fix/gradium-stt-interims
branch
from
October 16, 2026 08:23
0eea220 to
5fa1291
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Smaller fixes in the Gradium STT stream, each its own commit.
Interim transcripts carried one word instead of the partial utterance. Each
textpiece from the server was emitted as its ownINTERIM_TRANSCRIPT. The agent treats an interim as the whole current partial: it replaces the previous one for live captions, and when the final transcript is late it becomes the user's turn text. One-word interims made captions flicker and could hand the LLM only the last word. The stream now emits the joined buffer of the current utterance, with the utterance's start time, on both interim and final events.vad_bucket=0disabled end-of-turn detection. The option was tested for truthiness, so selecting bucket 0 (the 0.5 s horizon) meant noFINAL_TRANSCRIPTwas ever emitted. OnlyNonemeans disabled.No connect timeout.
ws_connectran under the shared aiohttp session's default of five minutes, so a hung handshake blocked the stream for minutes andconn_options.timeoutwas never applied. The connect is wrapped inasyncio.wait_for, and timeouts and handshake failures (401, 402, …) are mapped toAPITimeoutErrorandAPIStatusErrorso the base class retry rules apply instead of a raw aiohttp exception.update_optionsreconnected and lost the utterance in progress. Changingbuffer_size_secondsset the reconnect event; the receive task was cancelled without finalizing, so the words buffered so far were dropped (live, an update mid-sentence lost the first half of the transcript). Nothingupdate_optionschanges is read by the live session, so the stream now keeps its socket, and the reconnect loop that only this path drove is gone; the base class already retries a failed session.Websocket lifecycle hardening, matching the Deepgram, AssemblyAI and Soniox plugins: a 30 s heartbeat so a dead socket is noticed, transport
ERRORframes and failed audio writes surface as retryable connection errors instead of raw aiohttp exceptions, atextframe withoutstart_sno longer corrupts the turn, and handshake failures of any kind map to the base class error types.Testing
tests/test_plugin_gradium_stt.py: nine tests on the fake socket from the previous PR, one per fix; each fails without its commit. 19 tests in the module.ruff,ruff format --checkandmypy --strictclean on the plugin.api.gradium.aiwith the repository's 46 s speech sample: 122 interims, the longest 148 words, one final transcript.Stacked on #7870
The diff includes its commits until it merges; the last three commits are this PR. The connect-timeout commit touches the same two lines of
_connect_wsas #7815 (x-api-source), so whichever lands second needs a one-line rebase.🤖 Generated with Claude Code