fix(gradium): keep the TTS socket alive while the text source stalls (3/4) by hhugo-gradium · Pull Request #7817 · livekit/agents

hhugo-gradium · 2026-10-14T13:07:14Z

Summary

One socket-lifecycle fix for the Gradium TTS plugin, inside the connect helper the two streams share.

Keep the socket alive while the text source stalls. The receive timeout applied between any two server messages, and the server is silent while it waits for text, so an LLM pause longer than the timeout cut the reply with APITimeoutError and no retry, since audio had already been pushed. A client heartbeat at min(5 s, timeout / 2) makes the server's pongs reset aiohttp's receive timer; the cap keeps the pong ahead of the timer. The receive timeout is passed per receive() call rather than fixed at connect time.

Testing

tests/test_plugin_gradium_tts.py: the success test asserts the heartbeat value.
ruff, ruff format --check and mypy --strict clean on the plugin.
Live against api.gradium.ai: a stream with a 12 s gap between two text pushes and a 5 s receive timeout fails with APITimeoutError without this change and returns the full audio with it. Checked separately that an idle socket times out at the receive timeout without the heartbeat and stays open past it with one.

Stacked on #7672

#7815 (merged) made _connect_ws the single connect path with the handshake bounded, so this PR only adds the heartbeat and moves the receive timeout to the receive calls. The diff includes #7672's six commits until it merges; only the last commit is this PR.

🤖 Generated with Claude Code

devin-ai-integration

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Both TTS streams treated any websocket close as the end of the audio and logged server `{"type": "error"}` frames as an unknown message type. The server sends `end_of_stream` on success and an error frame followed by a close on failure, so a mid-sentence worker failure returned silently truncated audio with no error event and no retry, and a rejected setup (unknown voice, no credits) surfaced only as "no audio frames were pushed". Raise APIStatusError carrying the server's message and websocket close code for both an error frame and a close before end_of_stream, and re-raise APIError subclasses instead of wrapping them in APIConnectionError. Close codes 1002 (protocol error), 1003 (unsupported data: unknown model, setup validation) and 1008 (policy violation: credits, concurrency) are final; everything else stays retryable. The server reports unknown voice ids under 1011 like worker failures, so those still retry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

aiohttp reports reader and protocol failures (an oversized frame, a broken connection) as a WSMsgType.ERROR message and then CLOSED. Both receive loops skipped the ERROR message, so the actual exception in ws.exception() was lost and the failure showed up as a generic "closed unexpectedly". Raise APIConnectionError chained to it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Scripted fake websocket exercising both the one-shot and streaming TTS paths: audio plus end_of_stream succeeds and sends the expected setup, text and end_of_stream messages; an error frame raises APIStatusError with the server message and code, unwrapped; a close before end_of_stream raises with the close code; 1003/1008 are not retried while 1011 is; a transport error raises APIConnectionError chained to the socket's exception. Also pins the surrounding, pre-existing error paths the mapping now sits next to: a receive timeout raises APITimeoutError, a rejected handshake raises APIStatusError with the HTTP status and is not retried, a malformed audio frame raises APIConnectionError chained to the decode error, a close without a code and an error frame without fields are retryable, and a send failure mid-stream surfaces instead of hanging. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…y cap The org-level concurrency quota ("Concurrency limit exceeded: N active sessions") is reported as an error frame with code 1008, the same code as exhausted credits or an invalid API key. A slot frees when another session ends, so a retry can succeed there, and the frame carries nothing that tells the two apart. Treat 1008 as retryable; only 1002 (protocol error) and 1003 (unknown model, setup validation) stay final. Verified against api.gradium.ai: with 15 sessions open, the 16th gets an error frame with code 1008 and that message. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The base ChunkedStream retries a retryable error even after the attempt pushed audio, unlike SynthesizeStream. Now that a close before end_of_stream raises instead of truncating, a worker failure mid-reply would replay the start of the text after its audio was already played. An error after the first audio chunk is no longer retryable. Also correct the close-code comment: the server's idle and duration limits close with 1008 after an error frame, not with 1000/1001. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

When the socket closes before end_of_stream, the close frame's reason is the server's own explanation (e.g. "Insufficient credits to start session"), and the only one when no error frame came first. It was only in the error body; the message now ends with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…tial audio Drop the guard that marked an error non-retryable once a one-shot synthesis had pushed audio. The base ChunkedStream retries after partial audio on purpose and restarts under a fresh request_id (livekit#6346), which is what every other TTS plugin relies on; a provider-specific exception to that policy belongs in the base class, not here. The guard also fired on audio the emitter was still buffering, blocking retries that would have been safe. The test now checks the base behaviour: a retryable failure after the first chunk reconnects and completes on the second socket. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The receive timeout applied between any two server messages. After `ready` the server is silent until it has enough text, so an LLM that paused longer than the timeout mid reply made the stream fail with APITimeoutError, and since audio had already been pushed the base class did not retry: the utterance was cut. Send a client heartbeat at min(5s, timeout / 2); the server's pongs reset aiohttp's receive timer, and the cap keeps the pong ahead of it. The receive timeout is now passed per receive() call rather than fixed at connect time. Verified against api.gradium.ai: a stream with a 12s gap between two text pushes and a 5s receive timeout failed with APITimeoutError before and returns the full audio with this change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

devin-ai-integration

Devin Review found 1 new potential issue.

devin-ai-integration · 2026-10-21T08:23:49Z

retry loop


        except asyncio.TimeoutError:
            raise APITimeoutError() from None
+        except APIError:
+            raise
        except aiohttp.ClientResponseError as e:


🔴 Partial speech repeats after retry

When Gradium fails after sending audio, ChunkedStream retries the entire text and plays its beginning again. The voice pipeline forwards frames without checking their request IDs.

Learn more

A chunked TTS request sends its whole text in one WebSocket session. Once AudioEmitter has forwarded audio, the base retry loop closes the old emitter but cannot retract its frames; a retry opens a new session and synthesizes the whole text again. The voice pipeline forwards frames as they arrive, without using the fresh request ID to invalidate audio from the earlier attempt. The stream adapter also forwards both attempts' frames unchanged.

Example: Gradium sends 0.5 seconds of "hello world" and then closes with code 1011. The retry succeeds, so the listener hears the first 0.5 seconds twice before the rest of the sentence.

Recommended fix: Prevent retries after audio has been emitted from this one-shot stream, unless downstream playback can explicitly discard frames from an earlier request ID. Add an assertion on the actual sequence of emitted audio, not only the number of WebSocket connections.

Was this helpful? React with 👍 or 👎 to provide feedback.

hhugo-gradium · 2026-10-21T08:42:54Z

Closing: #7818 supersedes this. It replaces the same connect helper and receive loops, keeps a client heartbeat on the shared socket and drops the per-message receive timeout altogether, so the stall protection added here ships with it. #7818 is now stacked directly on #7672.

…socket instead The receive timeout applied between any two server messages. After `ready` the server is silent until it has enough text, so an LLM that paused longer than `conn_options.timeout` mid reply failed the stream with APITimeoutError, and with audio already out the base class did not retry: the reply was cut. Now only the wait for `ready` is timed; after it the session waits as long as the socket lives, and a client heartbeat (`WS_HEARTBEAT_S`) is how a dead socket is noticed. Replaces livekit#7817. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…socket instead The receive timeout applied between any two server messages. After `ready` the server is silent until it has enough text, so an LLM that paused longer than `conn_options.timeout` mid reply failed the stream with APITimeoutError, and with audio already out the base class did not retry: the reply was cut. Now only the wait for `ready` is timed; after it the session waits as long as the socket lives, and a client heartbeat (`WS_HEARTBEAT_S`) is how a dead socket is noticed. Replaces livekit#7817. The heartbeat is 10s: aiohttp closes the socket when a pong takes longer than half the interval, so a stall of the event loop (end-of-turn inference, GC) gets 5s before a socket that was fine is lost, and a dead transport is still found within 15s, well inside the server's 120s idle limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This was referenced Oct 14, 2026

feat(gradium): multiplex TTS replies over one shared websocket (2/2) #7818

Open

fix(gradium): identify the TTS plugin to the API with x-api-source (1/4) #7815

Merged

hhugo-gradium force-pushed the fix/gradium-tts-connect branch 4 times, most recently from 424d9cf to 2cb2239 Compare October 14, 2026 16:16

hhugo-gradium changed the title ~~fix(gradium): bound the TTS handshake and keep the socket alive on stalls~~ fix(gradium): keep the TTS socket alive while the text source stalls Oct 14, 2026

hhugo-gradium changed the title ~~fix(gradium): keep the TTS socket alive while the text source stalls~~ fix(gradium): keep the TTS socket alive while the text source stalls (3/4) Oct 15, 2026

hhugo-gradium marked this pull request as ready for review October 15, 2026 08:45

hhugo-gradium requested a review from a team as a code owner October 15, 2026 08:45

devin-ai-integration Bot reviewed Oct 15, 2026

View reviewed changes

hhugo-gradium force-pushed the fix/gradium-tts-connect branch 3 times, most recently from 7c7aea6 to 2bcba15 Compare October 19, 2026 09:19

hhugo-gradium mentioned this pull request Oct 19, 2026

fix(gradium): surface TTS server errors, close codes and transport errors (1/2) #7672

Merged

hhugo-gradium force-pushed the fix/gradium-tts-connect branch from 2bcba15 to 47d63fd Compare October 20, 2026 12:36

hhugo-gradium and others added 4 commits October 20, 2026 12:45

hhugo-gradium force-pushed the fix/gradium-tts-connect branch from 47d63fd to 3505999 Compare October 20, 2026 12:47

hhugo-gradium and others added 2 commits October 20, 2026 13:08

hhugo-gradium force-pushed the fix/gradium-tts-connect branch from 3505999 to b5b1be9 Compare October 20, 2026 13:10

This comment was marked as resolved.

Sign in to view

hhugo-gradium and others added 2 commits October 21, 2026 08:20

hhugo-gradium force-pushed the fix/gradium-tts-connect branch from b5b1be9 to 08b2069 Compare October 21, 2026 08:22

devin-ai-integration Bot reviewed Oct 21, 2026

View reviewed changes

hhugo-gradium closed this Oct 21, 2026

Navigation Menu

Repository navigation

fix(gradium): keep the TTS socket alive while the text source stalls (3/4) - #7817

fix(gradium): keep the TTS socket alive while the text source stalls (3/4)#7817
hhugo-gradium wants to merge 8 commits into
livekit:mainfrom
hhugo-gradium:fix/gradium-tts-connect

hhugo-gradium commented Oct 14, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot left a comment •

edited

Loading

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Oct 21, 2026

Uh oh!

hhugo-gradium commented Oct 21, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

hhugo-gradium commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Stacked on #7672

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

✅ Devin Review: No Issues Found

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

hhugo-gradium commented Oct 21, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

hhugo-gradium commented Oct 14, 2026 •

edited

Loading

devin-ai-integration Bot left a comment •

edited

Loading