fix(upliftai): sentence-chunked streaming TTS with pipelined synthesis, cancellation and prewarm - #6576
Open
zaidqureshi2 wants to merge 1 commit into
Open
Conversation
…s, cancellation and prewarm The streaming path buffered the entire LLM turn before sending any text to the server, so time-to-first-audio was full LLM generation plus synthesis. Rewritten to synthesize per sentence chunk (English + Urdu boundaries, 200 char cap) with up to 3 requests pipelined and audio emitted in order. Each chunk is now decoded in-plugin and emitted as raw PCM: a decoder shared across chunks truncates at MP3 file boundaries, and the SDK decoder is fail-open so this surfaced as silently missing speech. Chunks that decode to zero frames raise a retryable APIError. Also: cancel in-flight server requests on interruption, propagate server errors/disconnects/timeouts as API errors instead of silent truncation, per-attempt segments channel so 1.6.x retry replay works, native prewarm(), reconnection owned by the SDK retry loop instead of socket.io, and declare the previously-undeclared python-socketio dependency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
zaidqureshi2
force-pushed
the
upliftai/streaming-chunker
branch
from
July 28, 2026 00:01
01dade8 to
6fa6fd1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The UpliftAI plugin's
SynthesizeStreamdrained its token stream to completion before sending any text to the server, so no audio was synthesized until the LLM finished its entire turn — time-to-first-audio was full LLM generation time plus synthesis time. This PR rewrites the streaming path to synthesize per sentence chunk while the LLM is still streaming, and fixes several correctness issues found along the way.Streaming rewrite
۔,؟) plus full-width marks — once at leastmin_chunk_len(default 20) chars are buffered, or force-submitted atmax_chunk_len(default 200). Words are never split; both knobs are constructor params.multi-streamnamespace (per-request queues keyed by request id); audio is emitted strictly in submission order.AudioStreamDecoderand emitted as raw PCM. A decoder shared across chunks truncates at MP3 file boundaries (verified empirically — each request returns a complete file, andAudioEmitter's persistent decoder cannot span them). Since the SDK decoder is fail-open, this previously surfaced as silently missing speech.APIErrorinstead of leaving a silent gap.Correctness
cancelfor in-flight requests instead of letting them synthesize audio nobody hears.errormessages and disconnects propagate asAPIError/APIConnectionErrorthrough the audio queues (previously indistinguishable from successful completion); audio timeouts raiseAPITimeoutErrorinstead of silently truncating._runattempt so retry replay works, and the plugin no longer callsend_input()in itsfinally— a failed attempt stays eligible for retry under thepushed_duration == 0gate._mark_started()fires on the first chunk, so the TTFB metric measures first-chunk audio rather than end-of-turn.reconnection=Falseand any previous client is disconnected before replacement — recovery is owned by the SDK retry loop, and a stale auto-reconnecting client could flip connection state under a new one.prewarm(): background socket connect on agent activity start (strong task reference, deduplicated, cancelled inaclose()).Packaging
python-socketio[asyncio_client]dependency (previously imported but undeclared).Testing
tests/test_plugin_upliftai.py— 8 hermetic tests (fake socket client, no credentials or network): streaming starts before input ends with Urdu/English boundary splitting, server-error propagation, short-utterance flush, prewarm dedup and cancel-on-aclose, interruption cancels in-flight requests, MP3 per-chunk decode regression, and undecodable-chunk error surfacing.ruff check,ruff format,mypy(strict) andpytest --plugin upliftaiall pass.🤖 Generated with Claude Code