fix(gradium): surface TTS server errors, close codes and transport errors (1/2) by hhugo-gradium · Pull Request #7672 · livekit/agents

hhugo-gradium · 2026-10-08T16:03:29Z

Summary

Both Gradium TTS streams treated any websocket close as the end of the audio and logged server {"type": "error"} frames as an unknown message type. The server sends end_of_stream on success, and an error frame followed by a close on failure, so:

a mid-sentence worker failure returned silently truncated audio, with no error event and no retry;
a rejected setup (unknown voice, no credits, unknown model) surfaced only as the generic "no audio frames were pushed".

Three commits:

Raise on error frames and on a close before end_of_stream. Both raise APIStatusError carrying the server's message and the websocket close code, and APIError subclasses are re-raised instead of being wrapped in APIConnectionError. Close codes 1002 (protocol error) and 1003 (unsupported data: unknown model, setup validation) are final; everything else stays retryable under the base class rules. The server reports unknown voice ids under 1011 like worker failures, so those still retry.
Keep retrying on 1008. The org-level concurrency quota ("Concurrency limit exceeded: N active sessions") is reported with code 1008, the same as exhausted credits or a bad API key, and a slot frees when another session ends. The frame carries nothing that tells the cases apart, so 1008 stays retryable. Verified against the API: with 15 sessions open the 16th gets exactly that frame.
Surface transport errors. aiohttp reports reader failures as a WSMsgType.ERROR message; both loops skipped it and lost the exception. It is now raised as APIConnectionError chained to ws.exception().

Testing

New tests/test_plugin_gradium_tts.py (unit category, 21 tests): a scripted fake websocket drives both synthesize() and stream() through success, error frames with final and retryable codes including the concurrency cap, early close, and transport errors, asserting the exact exception type and body so that re-wrapping would be caught. It also pins the pre-existing error paths next to the change: receive timeout, rejected handshake (401, not retried), malformed audio frame, close without a code, error frame without fields, and a send failure mid-stream. 11 of the 21 fail on main; the 10 that pass there are the success paths, the untouched pre-existing paths, and the concurrency-cap retry, which main also happened to get right.
Review follow-ups: the close frame's reason, the server's own explanation, now ends the error message (its assertion fails without the change). A commit that stopped retrying a one-shot synthesize() after partial audio was added and then reverted on review: the base ChunkedStream retries after partial audio on purpose and restarts under a fresh request_id (fix(tts): restart ChunkedStream retries under a fresh request_id #6346), as every other TTS plugin relies on, so the plugin follows that policy; test_synthesize_is_retried_after_partial_audio pins it (a retryable failure after the first chunk reconnects and completes).
ruff check, ruff format --check and mypy --strict are clean on the plugin.
Live against api.gradium.ai: one-shot and streaming synthesis return audio and end on the server's end_of_stream; an invalid voice_id raises APIStatusError: Gradium returned error: Embeddings not found for ... where it previously produced no audio.

Stack

Based on main now that #7815 has merged. #7818 (multiplexing, which also carries the keep-alive heartbeat) is stacked on top of this one; #7817 is closed in its favour.

Both TTS streams treated any websocket close as the end of the audio and logged server `{"type": "error"}` frames as an unknown message type. The server sends `end_of_stream` on success and an error frame followed by a close on failure, so a mid-sentence worker failure returned silently truncated audio with no error event and no retry, and a rejected setup (unknown voice, no credits) surfaced only as "no audio frames were pushed". Raise APIStatusError carrying the server's message and websocket close code for both an error frame and a close before end_of_stream, and re-raise APIError subclasses instead of wrapping them in APIConnectionError. Close codes 1002 (protocol error), 1003 (unsupported data: unknown model, setup validation) and 1008 (policy violation: credits, concurrency) are final; everything else stays retryable. The server reports unknown voice ids under 1011 like worker failures, so those still retry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

aiohttp reports reader and protocol failures (an oversized frame, a broken connection) as a WSMsgType.ERROR message and then CLOSED. Both receive loops skipped the ERROR message, so the actual exception in ws.exception() was lost and the failure showed up as a generic "closed unexpectedly". Raise APIConnectionError chained to it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Scripted fake websocket exercising both the one-shot and streaming TTS paths: audio plus end_of_stream succeeds and sends the expected setup, text and end_of_stream messages; an error frame raises APIStatusError with the server message and code, unwrapped; a close before end_of_stream raises with the close code; 1003/1008 are not retried while 1011 is; a transport error raises APIConnectionError chained to the socket's exception. Also pins the surrounding, pre-existing error paths the mapping now sits next to: a receive timeout raises APITimeoutError, a rejected handshake raises APIStatusError with the HTTP status and is not retried, a malformed audio frame raises APIConnectionError chained to the decode error, a close without a code and an error frame without fields are retryable, and a send failure mid-stream surfaces instead of hanging. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…y cap The org-level concurrency quota ("Concurrency limit exceeded: N active sessions") is reported as an error frame with code 1008, the same code as exhausted credits or an invalid API key. A slot frees when another session ends, so a retry can succeed there, and the frame carries nothing that tells the two apart. Treat 1008 as retryable; only 1002 (protocol error) and 1003 (unknown model, setup validation) stay final. Verified against api.gradium.ai: with 15 sessions open, the 16th gets an error frame with code 1008 and that message. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The base ChunkedStream retries a retryable error even after the attempt pushed audio, unlike SynthesizeStream. Now that a close before end_of_stream raises instead of truncating, a worker failure mid-reply would replay the start of the text after its audio was already played. An error after the first audio chunk is no longer retryable. Also correct the close-code comment: the server's idle and duration limits close with 1008 after an error frame, not with 1000/1001. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

When the socket closes before end_of_stream, the close frame's reason is the server's own explanation (e.g. "Insufficient credits to start session"), and the only one when no error frame came first. It was only in the error body; the message now ends with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

devin-ai-integration

Devin Review found 2 new potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

hhugo-gradium · 2026-10-20T13:29:39Z

Live testing: this PR vs `main`

Same scenarios run against the real Gradium API on both heads: this PR at 1bd94e17e, main at 077f3dd9f. max_retry=2 and retry_interval=0.5 unless noted.

Scenario	`main`	this PR
`synthesize()` / `stream()`, normal	ok	ok, same latency and audio length
Connection cut after ~1 s of audio, `synthesize()`	returns "ok" with 1.1 s of a ~50 s reply	`APIStatusError(1006)`, not retried, 1 connection
Connection cut after ~1 s of audio, `stream()`	returns "ok" with 1.1 s	raises; the base class skips the retry because audio was already pushed
Unknown voice	`no audio frames were pushed for text: …`, 3 attempts	`Gradium returned error: Embeddings not found for …` (1011), 3 attempts
Unknown model	`no audio frames were pushed …`, 3 attempts	`Unknown model 'no-such-model'.` (1003), 1 attempt
Invalid API key	`no audio frames were pushed …`, 3 attempts	`Invalid or expired API key` (1008), 3 attempts
6 parallel requests, org limited to 3 sessions, `max_retry=0`	3 fail with `no audio frames were pushed …`	3 fail with `Concurrency limit exceeded: 3 active sessions` (1008, retryable)
Same, `max_retry=8`	6/6 ok	6/6 ok: the 1008 retries recover as slots free up
20 parallel requests on a capacity-limited deployment, `max_retry=8`	20/20 ok, 52 connections	19/20 ok, 52 connections; the one failure is `No worker currently available` (1011) after its retries ran out

The connection cut goes through a local TLS-passthrough proxy that aborts both sides after ~150 KB from the server.

Takeaways:

Dropped connections: on main, a connection lost mid-reply is reported as success with truncated audio. This PR raises instead, and neither path replays audio that was already played.
Error messages: every failure now carries the server's reason instead of the generic "no audio frames were pushed".
The 1008 retry: it recovers from a real concurrency limit as intended.
Known limitation: an invalid API key also closes with 1008, and an unknown voice with 1011, so both are still retried. Telling them apart needs the server to send the error class in the error frame; that is a follow-up on the Gradium side.
The 19/20 vs 20/20 difference: both heads made the same number of connections under the same retry policy, so it reflects worker availability at that moment, not a behavior change.

hhugo-gradium · 2026-10-20T13:32:04Z

@longcw, this is now ready

longcw

looks good to me. something nit:

…tial audio Drop the guard that marked an error non-retryable once a one-shot synthesis had pushed audio. The base ChunkedStream retries after partial audio on purpose and restarts under a fresh request_id (livekit#6346), which is what every other TTS plugin relies on; a provider-specific exception to that policy belongs in the base class, not here. The guard also fired on audio the emitter was still buffering, blocking retries that would have been safe. The test now checks the base behaviour: a retryable failure after the first chunk reconnects and completes on the second socket. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

hhugo-gradium · 2026-10-21T09:14:53Z

@longcw, I think it's ok now

hhugo-gradium force-pushed the fix/gradium-tts branch 2 times, most recently from c06bfaa to f7a0604 Compare October 14, 2026 12:34

hhugo-gradium changed the title ~~fix(gradium): TTS error handling, source header, voice selection, connect timeout~~ fix(gradium): surface TTS server errors, close codes and transport errors Oct 14, 2026

hhugo-gradium force-pushed the fix/gradium-tts branch from f7a0604 to 96340dc Compare October 14, 2026 12:48

hhugo-gradium marked this pull request as ready for review October 14, 2026 12:59

hhugo-gradium requested a review from a team as a code owner October 14, 2026 12:59

This comment was marked as resolved.

Sign in to view

This was referenced Oct 14, 2026

fix(gradium): keep the TTS socket alive while the text source stalls (3/4) #7817

Closed

feat(gradium): multiplex TTS replies over one shared websocket (2/2) #7818

Open

devin-ai-integration Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread livekit-plugins/livekit-plugins-gradium/livekit/plugins/gradium/tts.py

hhugo-gradium force-pushed the fix/gradium-tts branch from a4dc170 to 446d240 Compare October 14, 2026 16:00

hhugo-gradium changed the title ~~fix(gradium): surface TTS server errors, close codes and transport errors~~ fix(gradium): surface TTS server errors, close codes and transport errors (1/4) Oct 15, 2026

hhugo-gradium force-pushed the fix/gradium-tts branch 2 times, most recently from 862d384 to e80f97b Compare October 19, 2026 09:19

hhugo-gradium changed the title ~~fix(gradium): surface TTS server errors, close codes and transport errors (1/4)~~ fix(gradium): surface TTS server errors, close codes and transport errors (2/4) Oct 19, 2026

hhugo-gradium force-pushed the fix/gradium-tts branch from e80f97b to a139c47 Compare October 20, 2026 12:36

hhugo-gradium and others added 4 commits October 20, 2026 12:45

hhugo-gradium force-pushed the fix/gradium-tts branch from a139c47 to a7f6d2e Compare October 20, 2026 12:47

hhugo-gradium and others added 2 commits October 20, 2026 13:08

devin-ai-integration Bot reviewed Oct 20, 2026

View reviewed changes

Comment thread livekit-plugins/livekit-plugins-gradium/livekit/plugins/gradium/tts.py Outdated

Comment thread livekit-plugins/livekit-plugins-gradium/livekit/plugins/gradium/tts.py Outdated

longcw approved these changes Oct 21, 2026

View reviewed changes

Comment thread livekit-plugins/livekit-plugins-gradium/livekit/plugins/gradium/tts.py Outdated

hhugo-gradium changed the title ~~fix(gradium): surface TTS server errors, close codes and transport errors (2/4)~~ fix(gradium): surface TTS server errors, close codes and transport errors (1/2) Oct 21, 2026

longcw merged commit b277c56 into livekit:main Oct 22, 2026
17 checks passed

Navigation Menu

Repository navigation

fix(gradium): surface TTS server errors, close codes and transport errors (1/2) - #7672

fix(gradium): surface TTS server errors, close codes and transport errors (1/2)#7672
longcw merged 7 commits into
livekit:mainfrom
hhugo-gradium:fix/gradium-tts

hhugo-gradium commented Oct 8, 2026 •

edited

Loading

Uh oh!

CLAassistant commented Oct 8, 2026 •

edited

Loading

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

Uh oh!

Uh oh!

hhugo-gradium commented Oct 20, 2026

Uh oh!

hhugo-gradium commented Oct 20, 2026

Uh oh!

longcw left a comment

Uh oh!

Uh oh!

hhugo-gradium commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

hhugo-gradium commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Stack

Related

Uh oh!

CLAassistant commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hhugo-gradium commented Oct 20, 2026

Live testing: this PR vs main

Uh oh!

hhugo-gradium commented Oct 20, 2026

Uh oh!

longcw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hhugo-gradium commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hhugo-gradium commented Oct 8, 2026 •

edited

Loading

CLAassistant commented Oct 8, 2026 •

edited

Loading

devin-ai-integration Bot left a comment •

edited

Loading

Live testing: this PR vs `main`