Skip to content

feat(voice): speak assistant text while it streams - #135

Merged
johnmatthewtennant merged 11 commits into
mainfrom
jtennant/speak-streaming-assistant-text
Aug 21, 2026
Merged

feat(voice): speak assistant text while it streams#135
johnmatthewtennant merged 11 commits into
mainfrom
jtennant/speak-streaming-assistant-text

Conversation

@johnmatthewtennant

@johnmatthewtennant johnmatthewtennant commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Berd feeds incremental ACP assistant text into the active Pocket utterance instead of waiting for the response to finish. Speech begins once a complete sentence is available, and later text continues in the same utterance as it arrives.

Assistant text around a tool call remains one voice turn. Starting the tool flushes buffered speakable text, and each visible text block receives its own speaking and terminal delivery status. Internal synthesis chunks are not exposed as transcript blocks.

Reviewer-reproducible examples

  • With a model that emits incremental text, request a long response. Confirm speech begins before the full response has arrived and the transcript continues to grow.
  • Request a response containing assistant text before and after a tool call. Confirm the pre-tool text is delivered before the tool completes, post-tool text continues in the same voice turn, and both visible text blocks receive the appropriate delivery status.

@johnmatthewtennant
johnmatthewtennant marked this pull request as ready for review August 19, 2026 22:12
@johnmatthewtennant
johnmatthewtennant marked this pull request as draft August 20, 2026 01:43
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/speak-streaming-assistant-text branch 4 times, most recently from ac4ef5a to 46aabf3 Compare August 20, 2026 02:43
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/speak-streaming-assistant-text branch from 46aabf3 to 6968ad6 Compare August 20, 2026 11:44
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/speak-streaming-assistant-text branch 2 times, most recently from 2bfc91c to 7cfa1fb Compare August 20, 2026 12:05
@johnmatthewtennant
johnmatthewtennant marked this pull request as ready for review August 20, 2026 15:05
Comment thread src/features/voice-conversation/lib/nativeAssistantSpeech.ts Outdated
@loganj

loganj commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

🤖 Carl follow-up review at current head 35c78af9c21890f0c624d25971a55d75d722ae6e: the blocking status-owner defect from 62ab90b is addressed. This head clears my blocker.

The fix changes ActiveUtterance from one target to a deduplicated targets collection and retains the current utterance status (nativeAssistantSpeech.ts:15-24,99-104,259-291). When post-tool text arrives, its target is added; if playback already started, the new block immediately inherits speaking (:260-269). Every terminal path now projects status to all participating targets: failed, completed, interrupted event, and local barge-in/stop (:106-123,141-190). This resolves both failure modes from my original review: normal completion no longer marks only the pre-tool block, and interruption no longer attributes post-tool interruption solely to that first block.

The new regression (nativeAssistantSpeech.test.ts:233-291) exercises text → tool → text, verifies the post-tool block inherits speaking, then verifies both blocks become spoken on completion. The existing interruption test remains single-block; adding a cross-tool interruption assertion would make the regression proof symmetric, but the shared setUtteranceStatus() path is the same for completion, stream interruption, failure, and local interruption, so I do not consider that omission blocking.

Exact-head CI is green: frontend checks/unit tests, clean-room frozen install, Rust checks on macOS/Linux/Windows, transcript virtualization, frontend build smoke, and DCO (workflow run 32393632587; install run 32393632586). I inspected the exact head on fresh read-only Blox workstation carl-berd135-rereview2 (ID 1908348) and did not rerun broad suites.

Updated verdict: approve / ship 35c78af9c21890f0c624d25971a55d75d722ae6e.

@johnmatthewtennant
johnmatthewtennant requested a review from a team August 20, 2026 18:30
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/speak-streaming-assistant-text branch from 35c78af to ebfb49a Compare August 20, 2026 18:36
morgmart
morgmart previously approved these changes Aug 20, 2026
Base automatically changed from jtennant/stream-pocket-tts-audio to main August 21, 2026 00:50
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/speak-streaming-assistant-text branch from ebfb49a to 92fc2b2 Compare August 21, 2026 00:50

@morgmart morgmart left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated code review

Reviewed the complete exact three-dot comparison 6405382...92fc2b2 using static inspection only, after running the project code-review and user wes-review workflows. Final self-check covered every changed voice-streaming flow, accessibility, i18n, navigation/consent, async failure and never-completes behavior, lifecycle/races, test honesty, project/design-system rules, duplicate overlap, and blocking evidence/user effect. Found one blocking response-loss race. Supplied GitHub evidence was inspected: Frontend checks and unit tests is failing; macOS/Linux/Windows Rust checks, clean-room install, and DCO pass, while two downstream frontend jobs are skipped. Recommendation: REQUEST_CHANGES; required checks separately govern merge readiness.

Deterministic publication result: 1 blocking and 0 non-blocking finding(s) publishable; 1 duplicate(s) suppressed.

Comment thread src/features/voice-conversation/lib/nativeAssistantSpeech.ts
morgmart
morgmart previously approved these changes Aug 21, 2026

@morgmart morgmart left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated code review

Reviewed all nine changed files in the complete exact three-dot comparison 6405382...04f4874 using static inspection only, after explicitly running the project code-review and user wes-review workflows. The final self-check covered every changed voice-streaming flow; accessibility; i18n/localization; navigation and consent guards; async errors, never-completes behavior, lifecycle and races; test honesty; design-system and project laws; duplicate overlap; and concrete evidence plus user effect for blocking severity. No publishable findings remain. A same-inspection variant of reply loss is suppressed because it is the same underlying finishing-playback issue as a resolved automation thread with a substantive human-account reply. Supplied GitHub evidence is structurally valid and was inspected: Frontend checks and unit tests is failing, Windows Rust checks is in progress, macOS/Linux Rust checks, clean-room install, and DCO pass, and two downstream frontend jobs are skipped. Recommendation: APPROVE under the publication policy; required checks separately govern merge readiness.

Deterministic publication result: 0 blocking and 0 non-blocking finding(s) publishable; 1 duplicate(s) suppressed.

Completed checks currently reporting failure:

  • Frontend checks and unit tests: failure

Pending checks: 1 check(s) are not complete.

This approval reflects the completed code review only; merge readiness remains governed by the repository's required checks.

@morgmart morgmart left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated code review

Reviewed all nine changed files in the complete exact three-dot comparison 6405382...168968a using static inspection only, after explicitly running the project code-review and user wes-review workflows. The final self-check covered every changed voice-streaming flow; accessibility; i18n/localization; navigation and consent guards; async errors, never-completes behavior, lifecycle and races; test honesty; design-system and project laws; duplicate overlap; and concrete evidence plus user effect for blocking severity. No publishable findings remain. A same-inspection variant of reply loss is suppressed because it is the same underlying finishing-playback issue as a resolved automation thread with a substantive human-account reply. Supplied GitHub evidence is structurally valid and was inspected: DCO, transcript virtualization, frontend build smoke, frontend checks and unit tests, macOS/Linux/Windows Rust checks, and clean-room install all completed successfully. Recommendation: APPROVE under the publication policy; required checks separately govern merge readiness.

Deterministic publication result: 0 blocking and 0 non-blocking finding(s) publishable; 1 duplicate(s) suppressed.

Pending checks: 1 check(s) are not complete.

This approval reflects the completed code review only; merge readiness remains governed by the repository's required checks.

@johnmatthewtennant
johnmatthewtennant merged commit 9f27bbd into main Aug 21, 2026
8 checks passed
@johnmatthewtennant
johnmatthewtennant deleted the jtennant/speak-streaming-assistant-text branch August 21, 2026 10:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants