Skip to content

fix(upliftai): sentence-chunked streaming TTS with pipelined synthesis, cancellation and prewarm - #6576

Open
zaidqureshi2 wants to merge 1 commit into
livekit:mainfrom
uplift-initiative:upliftai/streaming-chunker
Open

fix(upliftai): sentence-chunked streaming TTS with pipelined synthesis, cancellation and prewarm#6576
zaidqureshi2 wants to merge 1 commit into
livekit:mainfrom
uplift-initiative:upliftai/streaming-chunker

Conversation

@zaidqureshi2

Copy link
Copy Markdown
Contributor

Summary

The UpliftAI plugin's SynthesizeStream drained its token stream to completion before sending any text to the server, so no audio was synthesized until the LLM finished its entire turn — time-to-first-audio was full LLM generation time plus synthesis time. This PR rewrites the streaming path to synthesize per sentence chunk while the LLM is still streaming, and fixes several correctness issues found along the way.

Streaming rewrite

  • Tokens are buffered and submitted at sentence boundaries — English and Urdu (۔, ؟) plus full-width marks — once at least min_chunk_len (default 20) chars are buffered, or force-submitted at max_chunk_len (default 200). Words are never split; both knobs are constructor params.
  • Up to 3 requests are pipelined ahead of playback over the multi-stream namespace (per-request queues keyed by request id); audio is emitted strictly in submission order.
  • Each chunk is decoded in-plugin with its own AudioStreamDecoder and emitted as raw PCM. A decoder shared across chunks truncates at MP3 file boundaries (verified empirically — each request returns a complete file, and AudioEmitter's persistent decoder cannot span them). Since the SDK decoder is fail-open, this previously surfaced as silently missing speech.
  • A chunk that yields zero frames from nonzero bytes raises a retryable APIError instead of leaving a silent gap.

Correctness

  • Interruption/teardown now sends the server a cancel for in-flight requests instead of letting them synthesize audio nobody hears.
  • Server error messages and disconnects propagate as APIError/APIConnectionError through the audio queues (previously indistinguishable from successful completion); audio timeouts raise APITimeoutError instead of silently truncating.
  • The segments channel is created per _run attempt so retry replay works, and the plugin no longer calls end_input() in its finally — a failed attempt stays eligible for retry under the pushed_duration == 0 gate.
  • _mark_started() fires on the first chunk, so the TTFB metric measures first-chunk audio rather than end-of-turn.
  • Socket.io clients are created with reconnection=False and any previous client is disconnected before replacement — recovery is owned by the SDK retry loop, and a stale auto-reconnecting client could flip connection state under a new one.
  • Native prewarm(): background socket connect on agent activity start (strong task reference, deduplicated, cancelled in aclose()).

Packaging

  • Declare the python-socketio[asyncio_client] dependency (previously imported but undeclared).

Testing

tests/test_plugin_upliftai.py — 8 hermetic tests (fake socket client, no credentials or network): streaming starts before input ends with Urdu/English boundary splitting, server-error propagation, short-utterance flush, prewarm dedup and cancel-on-aclose, interruption cancels in-flight requests, MP3 per-chunk decode regression, and undecodable-chunk error surfacing.

ruff check, ruff format, mypy (strict) and pytest --plugin upliftai all pass.

🤖 Generated with Claude Code

@zaidqureshi2
zaidqureshi2 requested a review from a team as a code owner July 27, 2026 23:55
…s, cancellation and prewarm

The streaming path buffered the entire LLM turn before sending any text to
the server, so time-to-first-audio was full LLM generation plus synthesis.
Rewritten to synthesize per sentence chunk (English + Urdu boundaries, 200
char cap) with up to 3 requests pipelined and audio emitted in order.

Each chunk is now decoded in-plugin and emitted as raw PCM: a decoder shared
across chunks truncates at MP3 file boundaries, and the SDK decoder is
fail-open so this surfaced as silently missing speech. Chunks that decode to
zero frames raise a retryable APIError.

Also: cancel in-flight server requests on interruption, propagate server
errors/disconnects/timeouts as API errors instead of silent truncation,
per-attempt segments channel so 1.6.x retry replay works, native prewarm(),
reconnection owned by the SDK retry loop instead of socket.io, and declare
the previously-undeclared python-socketio dependency.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@zaidqureshi2
zaidqureshi2 force-pushed the upliftai/streaming-chunker branch from 01dade8 to 6fa6fd1 Compare July 28, 2026 00:01

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant