[hhugo-gradium]

Summary

Every Gradium TTS reply opened its own websocket: TLS and websocket handshake, API-key resolution, then the session. The Gradium API supports several sessions on one socket (close_ws_on_eos: false, one client_req_id per session, every message and reply tagged with it), so the plugin now keeps one socket per TTS instance and runs every reply on it as a session, in parallel when the agent asks for parallel synthesis. This is the shape the ElevenLabs and Inworld plugins use for their multi-context connections.

• A reader task owns the socket and routes each frame to the session it is tagged with. Frames for a session nobody listens to any more are dropped; an error frame among them is logged, since it explains a close the live sessions are about to see; a malformed frame is skipped rather than taking every session down. When the socket dies or is closed, every open session is handed the error and the next session opens a new socket.
• prewarm() opens the socket when the agent activity starts, so the first reply skips the handshake too. A socket the server closed, after a failed session for instance, is reopened by the next reply.
• A session that does not complete, because the agent interrupted it, it timed out waiting for the server, or it failed, is cancelled with the API's cancel message: the server stops synthesizing it, releases the worker and answers cancelled as the session's last frame, so a retry on the same socket does not run next to an orphan holding a worker and a concurrency slot. The socket and the other sessions on it are unaffected, and whatever still arrives for the cancelled session is dropped. This is the same contract as ElevenLabs' close_context / isFinal and Inworld's close_context / contextClosed. Each session runs as a context manager on the connection (_Connection.session()) that sends its setup and, on an early exit, the cancel. The cancel goes out only once the setup did: the server answers a message for an id it has never seen with a 1002 protocol error and closes the socket under every other session. A cancel landing after the session ended on its own changes nothing on the server, so no bookkeeping tracks which sessions ended.
• Only the wait for the server's ready is bounded by conn_options.timeout. After that a session waits as long as the socket lives: the server is silent while it waits for text, and a dead socket is noticed through a client heartbeat on the shared socket, a fixed 10 s interval (aiohttp allows half of it for the pong, so an event loop stalled by inference or GC has 5 s before every session on the socket is lost); the server's pongs keep the socket alive while the text source stalls, and there is no per-message receive timeout left for the interval to stay ahead of. This folds in what fix(gradium): keep the TTS socket alive while the text source stalls (3/4) #7817 did on the single-socket design; that PR is closed in favour of this one.
• Two things other multi-context plugins do are left out on purpose. There is no per-socket cap on sessions (Inworld caps at five and opens more sockets): the server enforces the org's concurrency, and a reply it refuses with 1008 retries on the same socket rather than opening a second one. And there is no per-session timeout after ready (ElevenLabs re-arms a timer on every audio frame): the heartbeat and the server's 120 s idle limit are the bounds, since a timeout between frames is exactly what cut replies behind a slow LLM.
• aclose() fails the in-flight sessions, closes the streams and the socket. The setup message is built once for both streams, and ChunkedStream no longer carries arguments it does not use.

Server dependency

Two server changes have to be live before this PR is released:

• The cancel message, gradium-serve export session state as db and compute delta for sync #4604, merged and in api 0.97.1. Against an older server a cancel fails message validation and the server closes the socket with 1011, which the plugin reports as a retryable connection error on the live sessions; the next reply reconnects.
• gradium-serve feat(eot): add audio models AGT-2520 #4722, open. Without it a session that the satellite forwards to a peer keeps its org concurrency slot until the socket closes, so a long-lived shared socket runs out of slots after max_concurrent_websockets replies and every later setup is refused with "Concurrency limit exceeded". That happens on prod whenever no local worker is free.

Both verified live on staging on 2026-10-20 with this branch: one socket, three replies, an interrupted reply acknowledged with cancelled, a reply after the interruption and one after 30 s idle, on a satellite serving locally and on one forwarding to a peer. Everything else here works against the current API.

Testing

• tests/test_plugin_gradium_tts.py: the fake socket carries any number of sessions, one script per setup it receives, tagging frames with that session's id, acknowledging a cancel with cancelled, and blocking while idle like a live socket. Fifteen tests for the shared connection: sequential and parallel replies share one socket with distinct ids; frames and error frames of unknown sessions are dropped or logged; an interrupted reply, streaming or one-shot, sends cancel, its later frames are dropped without a warning and its sibling completes on the same socket; a failed session raises for that session and the next reply reconnects after the server closes the socket; a connection loss reaches every open session; prewarm() connects ahead of the first reply; a server close or reader failure while idle reconnects without a retry; the wait for ready times out and silence after ready does not; aclose() closes the socket and ends an in-flight one-shot instead of hanging it; a malformed frame is skipped; a session that timed out is cancelled server-side and a completed one is not. A review pass on the shared connection added four more, folded into the commits they cover: a session that already ended with an error frame is not sent a cancel (the server would not acknowledge it) and leaves no bookkeeping behind; a cancelled frame nobody asked for ends the session with a retryable error instead of hanging it; a close code survives the window in which aiohttp has marked the socket closed but the reader has not recorded the CLOSE frame yet, so a 1003 stays final; an error frame with no client_req_id fails every open session and the next reply reconnects. A second review pass after the rebase added three more: aclose() closes a socket whose connect was still in flight; a send that fails with a connection error drops the socket so the retry opens a new one, for both streams. The restructure into _Connection.session() added one more, for both streams: a reply interrupted while its setup is still on its way out sends no cancel and leaves the socket usable. 68 tests across the Gradium modules at this head, all passing; every test closes its TTS so no task leaks.
• ruff, ruff format --check and mypy --strict clean on the plugin.
• Live against api.gradium.ai, everything except the cancel path, which needs export session state as db and compute delta for sync #4604:

Scenario	Result
five sequential replies then 30 s idle, prewarmed	one socket throughout; first audio 55 to 105 ms
three requests started together on one TTS	one socket; first audio 108, 110 and 111 ms
same five-reply sequence on the single-use code (#7672 head)	six sockets; first audio 110 to 146 ms

Re-run on 2026-10-20 against api.gradium.ai once every prod satellite was on api 0.97.1, which carries #4604:

Scenario	Result
prewarm, three replies, an interrupted reply, a reply right after, a reply after 30 s idle (twice)	one socket each run; the cancel acknowledged with cancelled within 0.5 s; first audio 55 to 118 ms, one outlier at 533 ms
three replies started together, one interrupted at 0.5 s of audio, then a fourth	one socket; siblings unaffected, 6.3 s of audio each; cancel acknowledged; first audio 160 ms in parallel, 97 ms alone

The same sequence also passed on staging through a satellite that forwards every session to a peer.

Agent-shaped run on 2026-10-22 against api.gradium.ai, at this head: a real LLM (LiveKit Inference, gpt-4.1-mini) streams each reply token by token into tts.stream(), as AgentSession does. Prewarm, two turns, a barge-in at 1.5 s of audio, the next turn right away, 30 s idle, a turn, then a filler and a reply synthesized together. Four runs each on this branch and on main's plugin, alternating.

	This branch	main
sockets per conversation	1 in every run; every cancel acknowledged with cancelled within 20 ms, no error frames	7, one per reply
streamed turns, first audio after the LLM's first token (20 turns)	median 135 ms, p90 166	median 139 ms, p90 159
two one-shot replies started together, first audio (8 replies)	median 122 ms	median 161 ms

With an LLM in front, main's handshake is hidden behind the first token, so streamed turns do not get faster; replies that do not wait on an LLM (fillers, say()) skip the handshake and start about 40 ms earlier.

Commits, in review order

1. refactor(gradium): build the TTS setup message once, drop unused ChunkedStream arguments: _setup_msg replaces the two inline setup dicts; the one-shot stream stops taking the API key, endpoint and model name it did not use.
2. refactor(gradium): share the receive loop of both TTS streams: the two identical receive loops become one _consume_session(ws, on_audio=...); the callers keep their flush() / end_segment().
3. test(gradium): a TTS test harness that behaves like a live socket: test-only. The fake blocks while idle, takes one script per session and can queue unsolicited frames; _drain returns the audio duration; every TTS is closed after its test.
4. refactor(gradium): tag every TTS session message with a client_req_id: one id per reply, on setup, text and end_of_stream, also used as the emitter's request or segment id; _run_ws becomes _run_session. Nothing reads the id yet.
5. fix(gradium): time only the wait for ready on TTS sessions, ping the socket instead: what fix(gradium): keep the TTS socket alive while the text source stalls (3/4) #7817 fixed, the way this PR needs it. Only the wait for ready is bounded by conn_options.timeout; after it a 10 s client heartbeat is the liveness check, so a stalled LLM no longer cuts the reply.
6. feat(gradium): share one websocket per TTS instance across replies: the core. _Connection with the reader task and per-session routing, close_ws_on_eos: false, _Connection.session() as the context manager each reply runs through (registers the session, sends its setup, stops listening on exit), _consume_session reading from the session's queue, _connection() under a lock, prewarm(), TTS.aclose() under the same lock, a failed send failing the connection, and an aclose() that keeps the server's close code when aiohttp has already marked the socket closed; the shared-connection tests.
7. feat(gradium): cancel interrupted TTS sessions server-side: the cancel on an early exit of session(), sent only once the setup went out; an unrequested cancelled ends the session; the cancel tests, including the one for a reply interrupted while its setup is still on its way out.

Every commit passes the Gradium tests on its own. The branch was squashed from fourteen commits on 2026-10-22, with the tree unchanged: the review fix-ups now sit in the commits they correct.

The agent-shaped live run above was repeated at this head on 2026-10-22: one socket per conversation, the barge-in acknowledged with cancelled within 10 ms, first audio after the first token median 128 ms, one-shot median 122 ms.

History

#7672 (1/2) has merged and this branch is rebased on main: the diff is only this PR's commits. #7815 has merged and #7817 is closed, its heartbeat lives here.

🤖 Generated with Claude Code

[devin-ai-integration]

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 5 potential issues.

[devin-ai-integration]

🟡 Prewarm stalls short-timeout replies

When prewarm() holds _conn_lock during its handshake, a reply waits there beyond its configured timeout. _connect_ws only applies the reply's timeout after the lock becomes available.

Learn more

The lock serializes socket creation, and prewarm uses the default connection timeout of 10 seconds. A reply's shorter timeout is passed to _connect_ws only if that reply reaches the new handshake; waiting for the lock and closing a stale connection do not count toward that limit. Thus an already-started prewarm can hold a time-sensitive reply for much longer than its connection timeout.

Example: prewarm() starts a handshake that takes 8 seconds. A reply using APIConnectOptions(timeout=0.2) starts immediately afterward. It waits almost 8 seconds for _conn_lock, then uses the prewarmed socket, rather than failing after 0.2 seconds.

Recommended fix: Bound the entire _connection acquisition with the caller's deadline, including lock wait and stale-connection cleanup. Ensure that timing out a waiting reply does not cancel another caller's prewarm or connection attempt.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟡 Pre-ready frames prevent setup timeout

If non-ready frames keep arriving, _consume_session restarts ready_timeout on every frame. A reply can wait indefinitely for ready on an otherwise live socket.

Learn more

A session starts a countdown to the server's ready frame. The current code creates a new wait_for on every queue read until ready arrives. Receiving any other recognized or unknown frame resets the countdown. Because the websocket remains live, the heartbeat cannot end this wait.

Example: With timeout=0.2, a server sends need_more_text frames every 0.1 seconds but never sends ready. The reply continues waiting past 0.2 seconds instead of raising APITimeoutError.

Recommended fix: Calculate a monotonic deadline when beginning the ready wait, and pass only its remaining duration into subsequent queue reads. Keep the unbounded wait only after ready is received.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟨 Provider error escapes log redaction

When an earlier session fails, _read logs its provider-supplied message under the unmarked error key. Messages containing sensitive provider content bypass PII attribute redaction.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 1 new potential issue.

[devin-ai-integration]

🔴 Partial speech replays after socket failure

When Gradium emits audio before a retryable socket failure, ChunkedStream retries the entire text. Voice generation forwards already emitted frames without checking the new request ID, so listeners hear repeated speech.

Learn more

The chunked TTS base class creates a fresh emitter for each retry, but it does not remove audio already read from the stream. The removed guard let an error retry only when the first attempt had pushed no audio. After the guard is gone, audio from the failed attempt can reach the listener before the retry emits the same words again. StreamAdapterWrapper and voice generation forward frames without using the changed request ID to discard the stale attempt.

Example: A provider sends 0.5 seconds of "Hello world," pauses, and closes with code 1011. The caller hears the first 0.5 seconds, then hears the start again when the full synthesis retries.

Recommended fix: Either prevent one-shot retries after audio has been made available to consumers, or implement attempt-aware replacement through the entire playback path before allowing partial-audio retries. Test with a pause between the first audio frame and the failure so the first attempt's frames actually reach the caller.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 1 new potential issue.

[devin-ai-integration]

🟡 Send failure masks terminal server close

When a 1003 close races a send, send can fail before the reader records the close code. _fail then gives every session a retryable error, so terminal rejections are retried.

Learn more

A shared connection routes server frames to active synthesis sessions. aiohttp can reject a write after it begins closing the transport, before the reader has recorded the CLOSE frame and its code. Here _fail stores a generic retryable error in every session queue. The later reader error cannot replace it because _fail returns when _error is set. This loses non-retryable close codes for sessions that have not yet consumed their queues, including siblings of the failed send.

Example: The server rejects an invalid model with code 1003 while another reply sends text. Its write raises ClientConnectionResetError before the reader processes CLOSE. Both replies receive a retryable connection error instead of the terminal 1003 status, so their retry loops can reconnect with the same invalid model.

Recommended fix: Preserve the pending reader's close status when a failed send overlaps a server close, as _Connection.aclose does for a closed websocket. Keep immediate failure for a genuine dead transport, but avoid committing a generic error over an available terminal close code; add a test with a pending 1003 CLOSE and a concurrent failed send.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Devin Review found 1 new potential issue.

[devin-ai-integration]

🔴 Interrupted setup leaves server workers occupied

If interrupted while send_str drains a written setup frame, setup_sent stays false and skips cancellation. The server keeps synthesizing the abandoned session and holds its concurrency slot until timeout.

Learn more

An aiohttp WebSocket send writes the frame to its transport before it awaits flow-control drain. Cancellation during that drain leaves the send unfinished even though the setup can reach the server. The context manager then skips cancel because it marks setup as sent only after the await returns. The abandoned server session holds a worker and concurrency slot until the server's idle timeout. This affects both ChunkedStream and SynthesizeStream.

Example: A streaming reply begins setup on a backpressured socket. aiohttp writes the setup frame, then waits for the transport to drain. The agent interrupts that reply during the wait. No cancel is sent, although the server has started the session.

Recommended fix: Make setup delivery and cancellation atomic with respect to task interruption, for example by allowing the send to finish under shielding before deciding whether to cancel. If delivery cannot be determined, retire the socket so a potentially delivered setup cannot retain a session on the shared connection. Preserve the no-cancel behavior when setup definitely was not sent.

---

Was this helpful? React with 👍 or 👎 to provide feedback.