Skip to content

fix(gradium): STT cumulative interims, vad_bucket=0, connect timeout, websocket hardening (2/2) - #7871

Draft
hhugo-gradium wants to merge 9 commits into
livekit:mainfrom
hhugo-gradium:fix/gradium-stt-interims
Draft

hhugo-gradium wants to merge 9 commits into
livekit:mainfrom
hhugo-gradium:fix/gradium-stt-interims

Conversation

@hhugo-gradium

@hhugo-gradium hhugo-gradium commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Smaller fixes in the Gradium STT stream, each its own commit.

  • Interim transcripts carried one word instead of the partial utterance. Each text piece from the server was emitted as its own INTERIM_TRANSCRIPT. The agent treats an interim as the whole current partial: it replaces the previous one for live captions, and when the final transcript is late it becomes the user's turn text. One-word interims made captions flicker and could hand the LLM only the last word. The stream now emits the joined buffer of the current utterance, with the utterance's start time, on both interim and final events.

  • vad_bucket=0 disabled end-of-turn detection. The option was tested for truthiness, so selecting bucket 0 (the 0.5 s horizon) meant no FINAL_TRANSCRIPT was ever emitted. Only None means disabled.

  • No connect timeout. ws_connect ran under the shared aiohttp session's default of five minutes, so a hung handshake blocked the stream for minutes and conn_options.timeout was never applied. The connect is wrapped in asyncio.wait_for, and timeouts and handshake failures (401, 402, …) are mapped to APITimeoutError and APIStatusError so the base class retry rules apply instead of a raw aiohttp exception.

  • update_options reconnected and lost the utterance in progress. Changing buffer_size_seconds set the reconnect event; the receive task was cancelled without finalizing, so the words buffered so far were dropped (live, an update mid-sentence lost the first half of the transcript). Nothing update_options changes is read by the live session, so the stream now keeps its socket, and the reconnect loop that only this path drove is gone; the base class already retries a failed session.

  • Websocket lifecycle hardening, matching the Deepgram, AssemblyAI and Soniox plugins: a 30 s heartbeat so a dead socket is noticed, transport ERROR frames and failed audio writes surface as retryable connection errors instead of raw aiohttp exceptions, a text frame without start_s no longer corrupts the turn, and handshake failures of any kind map to the base class error types.

Testing

  • tests/test_plugin_gradium_stt.py: nine tests on the fake socket from the previous PR, one per fix; each fails without its commit. 19 tests in the module.
  • ruff, ruff format --check and mypy --strict clean on the plugin.
  • Live against api.gradium.ai with the repository's 46 s speech sample: 122 interims, the longest 148 words, one final transcript.

Stacked on #7870

The diff includes its commits until it merges; the last three commits are this PR. The connect-timeout commit touches the same two lines of _connect_ws as #7815 (x-api-source), so whichever lands second needs a one-line rebase.

🤖 Generated with Claude Code

hhugo-gradium and others added 9 commits October 16, 2026 06:21
The STT stream looked up `delay_in_tokens` in the ready message, a key
the server never sends (it is `delay_in_frames`), so the hard-coded
fallback of 6 steps was always used. Production models run with a delay
of 10, so a turn was finalized 4 steps (320ms) before the model had
emitted its last words, which then showed up as a spurious new turn.

Adds a scripted fake websocket harness for the STT tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
When the input channel closed, send_task returned without telling the
server; `closing_ws` was never set, so recv_task polled until the server
dropped the idle socket, which was then reported as an unexpected close
and retried against a closed input. The unused `_CLOSE_MSG`
(`terminate_session`) was another provider's protocol.

Send `{"type": "end_of_stream"}` once the input ends, mark the socket as
closing, and on the server's end_of_stream emit the buffered words as
the final transcript before returning. Raise APIError with the server's
message on `error` frames instead of logging them as unknown. Skip the
VAD silence flush once the input channel is closed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
They only talk to fake websockets, so they belong in the hermetic
category that CI runs rather than the provider category.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Only the server's end_of_stream path emitted the words buffered since
the last VAD finalization. The two other clean exits after end_input(),
a close frame instead of the reply and the receive timeout, dropped them
without a FINAL_TRANSCRIPT. Finalize on all three.

A server-initiated end_of_stream while audio is still being sent (a
session duration limit, for instance) was treated as a clean end: the
receive task returned and the next audio write failed with a raw
connection error the base class does not retry. Finalize, then raise a
retryable APIStatusError so the stream reconnects.

The final end_of_stream send is guarded: if the server closed the
socket at the same moment, that is not an error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Each `text` piece from the server was emitted as its own
INTERIM_TRANSCRIPT. The agent treats an interim as the whole partial
utterance: it replaces the previous one for live captions and, when the
final transcript is late, becomes the user's turn text. One-word interims
made captions flicker and could hand the LLM only the last word.

Emit the joined buffer of the current utterance, with the utterance's
start time, on both interim and final events.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`vad_bucket` was tested for truthiness, so selecting bucket 0 (the 0.5s
horizon) silently disabled end-of-turn detection and no FINAL_TRANSCRIPT
was ever emitted. Only None means disabled.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ws_connect ran under the shared aiohttp session's default timeout of
five minutes, so a hung handshake blocked the stream for minutes and
conn_options.timeout was never applied. Wrap the connect in
asyncio.wait_for and map timeouts and handshake failures (401, 402, ...)
to APITimeoutError and APIStatusError so the base class retry rules
apply instead of a raw aiohttp exception.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Bring the STT stream in line with the other websocket STT plugins:

- Map every connect failure, not only timeouts and HTTP rejections, to
  APIConnectionError: a DNS failure, a refused connection or a TLS error
  raised an aiohttp exception the base class treats as fatal on the
  first attempt.
- Open the socket with a 30s heartbeat. Without it a silently dropped
  socket was never noticed: the receive loop polled forever and audio
  kept going into a dead socket, with no error and no reconnect.
- Handle the ERROR frame the heartbeat produces, instead of logging it
  as an unexpected type and losing the reason.
- Guard the audio writes: a mid-write socket drop raised a raw
  connection error; it is now a retryable APIConnectionError, or a quiet
  return when the stream was closing anyway.
- Read a text frame's fields before flipping the speaking state, so a
  malformed frame cannot leave the turn started without a
  START_OF_SPEECH.
- Drop the unused `_speech_duration` and `bytes_per_frame`, align the
  STTOptions vad_threshold default with the constructor's, and fix the
  stale "only final transcripts" comment.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`update_options(buffer_size_seconds=...)` set the reconnect event, so the
stream tore down its websocket and opened a new one. The receive task was
cancelled without finalizing, which dropped the words buffered for the
utterance in progress: live, an update mid-sentence lost the first half
of the transcript (WER 0.58 against 0.11 for the same audio).

Nothing `update_options` changes is used by the live session
(`buffer_size_seconds` is only stored), so there is nothing to reconnect
for. Keep the socket, and drop the reconnect loop that only this path
drove; the base class already retries a failed session.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@hhugo-gradium
hhugo-gradium force-pushed the fix/gradium-stt-interims branch from 0eea220 to 5fa1291 Compare October 16, 2026 08:23
@hhugo-gradium hhugo-gradium changed the title fix(gradium): STT cumulative interims, vad_bucket=0, connect timeout (2/2) fix(gradium): STT cumulative interims, vad_bucket=0, connect timeout, websocket hardening (2/2) Oct 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant