Repository navigation
fix(gradium): identify the STT plugin to the API with x-api-source - #2750
rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 6e94213 The changes in this PR will be included in the next version bump. This PR includes changesets to release 42 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 4 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| async #sendAudio(ws: WebSocket, signal: AbortSignal): Promise<void> { | ||
| const audio = new AudioByteStream(this.#opts.sampleRate, 1, DEFAULT_FRAME_SIZE); |
There was a problem hiding this comment.
🔴 Buffered speech disappears on reconnect
When a socket reconnects, #sendAudio discards audio consumed but not yet assembled into a full frame. Speech crossing that boundary loses its first buffered samples.
Learn more
AudioByteStream collects input until it has 1920 samples. The base SpeechStream retries run() with the same input queue, but this method constructs a new buffer for each attempt. Audio consumed into an incomplete frame cannot be replayed on the new socket.
Example: A 40 ms audio frame enters the stream, then the socket closes. The frame stays in the old 80 ms byte buffer; after reconnect, the next 40 ms starts a new buffer instead of completing the original frame.
Recommended fix: Keep the byte buffer across connection attempts, or requeue untransmitted samples before retrying. Validate behavior with a disconnect after a partial chunk.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if (signal.aborted) return; | ||
| throw error; | ||
| } | ||
| if (result.done) return; |
There was a problem hiding this comment.
🔴 Trailing speech lost at input completion
When endInput() closes the input queue, #sendAudio returns without flushing its partial frame. The provider never receives speech shorter than the remaining 80 ms buffer.
| if (result.done) return; | |
| if (result.done) { | |
| for (const frame of audio.flush()) this.#sendFrame(ws, frame.data); | |
| return; | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
| // Input completion does not end a Gradium session. Keep receiving final | ||
| // provider events, while still propagating a sender failure immediately. | ||
| const senderFailure = sender.then( | ||
| () => new Promise<never>(() => {}), | ||
| (error: unknown) => Promise.reject(error), | ||
| ); | ||
| await Promise.race([receiver, senderFailure]); | ||
| if (!this.abortSignal.aborted) { | ||
| throw new APIConnectionError({ | ||
| message: 'Gradium connection closed unexpectedly', | ||
| options: { retryable: true }, | ||
| }); |
There was a problem hiding this comment.
🔴 Completed speech streams never settle normally
After endInput(), run keeps waiting for the provider to close its socket. It treats that close as failure, so consumers wait indefinitely or receive a retry error.
Learn more
The base endInput closes only the audio input queue. #sendAudio returns, but senderFailure deliberately never settles on success. When the provider closes after processing the last audio, the receiver resolves and run() throws a retryable error instead of completing. A provider that keeps connections open leaves the stream pending forever.
Example: A caller pushes one utterance, calls endInput(), and iterates the stream to completion. Even after the provider finishes, iteration hangs if its socket stays open, or ends with a connection error if it closes.
Recommended fix: Define an input-completion protocol: send the provider's end-of-audio signal, drain its final responses, and return successfully on the corresponding completion event or expected close.
Was this helpful? React with 👍 or 👎 to provide feedback.
| this.#speaking = false; | ||
| this.#bufferedText = []; | ||
| this.#remainingVadSteps = undefined; |
There was a problem hiding this comment.
🔴 Active transcripts vanish on socket loss
If a socket closes before VAD finalization, run clears #bufferedText during retry. Interim speech never becomes a final transcript, so the agent loses that utterance.
Learn more
The text handler emits only interim events and keeps words in #bufferedText. Final events require enough inactivity step messages. A network drop before that threshold restarts run(), clearing that state and preventing the partial utterance from ever becoming final.
Example: Gradium sends text: "hello" and then drops the connection before sending a VAD step. The stream retries and clears hello; the agent never receives a final hello transcript.
Recommended fix: Finalize or preserve pending transcript state before reconnecting, with an end-of-speech boundary when appropriate.
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
x-api-source: livekit-pluginThe target did not have a Gradium plugin counterpart, so this PR includes the package, streaming STT transport, workspace registration, API report, and release changeset required to preserve the source behavior. No source-dependent behavior was dropped.
Source diff coverage
livekit-plugins/livekit-plugins-gradium/livekit/plugins/gradium/api.pytoplugins/gradium/src/api.ts: preserves the shared API source constant and websocket header helper in TypeScript.livekit-plugins/livekit-plugins-gradium/livekit/plugins/gradium/stt.pytoplugins/gradium/src/stt.ts: the target had no Gradium counterpart, so the Gradium streaming STT infrastructure was added and its websocket connection delegates headers to the shared helper. Python seconds-based buffering is exposed as target-conventionalbufferSizeMs.tests/test_plugin_gradium_api_source.pytoplugins/gradium/src/api_source.test.ts: uses a local Vitest websocket server, pinslivekit-pluginrather than importing it, and asserts bothx-api-keyandx-api-sourceupgrade headers.Testing
pnpm test plugins/gradiumpnpm --filter @livekit/agents-plugin-gradium typecheckpnpm --filter @livekit/agents-plugin-gradium buildpnpm --filter @livekit/agents-plugin-gradium api:checkpnpm buildpnpm lint(passes with existing repository warnings)pnpm format:checkSource: livekit/agents#7873