feat(desktop): real-time voice dictation in composer#1511
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c12132e30
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 195d741e65
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e874a53dbf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
klopez4212
force-pushed
the
kennylopez-dictation
branch
from
e874a53 to
ebcd42e
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ebcd42e0b5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b9cc335a8e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
klopez4212
force-pushed
the
kennylopez-dictation
branch
from
b9cc335 to
422b855
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 422b8556b5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7af4eeebe8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d498da598
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6d4f2ebbe2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5ce09acd75
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
wesbillman
left a comment
wesbillman
left a comment
There was a problem hiding this comment.
Review (Brain, agent review on Wes's behalf) — solid architecture; three questions/asks before merge
The core design is right: relay holds the key and mints short-lived client secrets; both /transcribe/* endpoints reuse the bridge's NIP-98 verify + replay guard + relay-membership enforcement (I traced authenticate() against bridge.rs — it's the same tenant-bound expected-URL construction, so the auth story is consistent). Graceful degradation via /transcribe/status → hidden mic button is clean, and the stale-run token in useRealtimeDictation handles the stop-during-setup races carefully. The transcript-merge item-ordering tests (out-of-order completions, previous_item_id insertion) are the right coverage.
Blocking-ish questions:
-
No rate limit on
POST /transcribe/session. Any relay member can mint unlimited ephemeral OpenAI secrets — each of which opens a metered Realtime session on the relay operator's bill. The hooks endpoint and media uploads have abuse considerations; this endpoint spends the operator's money directly. At minimum a per-pubkey cooldown or a simple token bucket seems warranted before this ships. What's the plan? -
Auto-submit "submit" is hardcoded and always on (
DEFAULT_AUTO_SUBMIT_PHRASE, no setting, no opt-out). Saying the word "submit" at the end of an utterance sends the message immediately — in a work chat, dictating "…we should submit" fires a send.parseAutoSubmitPhrasesis clearly built for user-configurable phrases; wiring it to a constant makes this a surprising default. Suggest shipping with auto-submit off (or behind a setting) and keeping stop-then-review as the default flow. -
Unrelated e2e spec modified to keep passing (
human-edit-agent-content.spec.ts:input.clear()/fill()→ select-all + type, plus a not-empty wait). This means the composer's edit-mode DOM behaviour observably changed under this PR — presumably from thesetEditorContentRefwiring. What exactly changed, and isinput.fill()now broken for the composer generally (which would affect other tests/tooling), or just slower to populate?
Non-blocking:
reqwest::Client::new()per request increate_transcribe_session— connection pooling is lost; aOnceLock/state-held client is the usual pattern. Fine at dictation call rates, just noting.transcription_model()readsBUZZ_TRANSCRIPTION_MODELfrom env at request time instead ofConfig::from_envlike every other knob (including the API key one line away). Worth moving intoConfigfor consistency.int16ToBase64builds the binary string with per-byte+=— fine for 20ms frames, would be a hotspot if chunk sizes grow.- Buffer cap silently drops audio after ~10s of pre-connect buffering (
MAX_BUFFER_CHUNKS); worst case the user talks through a very slow session setup and loses tail audio with no signal. Rare, but a toast on overflow would be cheap.
CI is fully green including relay e2e and Windows.
|
Thanks for the thorough review @wesbillman — addressing the three blocking questions: 1. No rate limit on Fair point — each session mints a metered OpenAI Realtime session on the relay operator's bill. I'll add a per-pubkey token bucket (e.g. 5 sessions/minute, 30/hour) to the endpoint before merge. The auth pipeline already extracts the pubkey so it's straightforward to layer a rate limiter on top. 2. Auto-submit always on Agreed this is surprising as a default. I'll ship with auto-submit disabled by default and gate it behind a user setting (the 3. Unrelated e2e spec change ( This wasn't caused by the dictation wiring — it's a pre-existing Playwright/Tiptap interaction issue. Also pushed Will follow up with commits for #1 (rate limit) and #2 (auto-submit off by default) shortly. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 900f37d920
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…fault Addresses Wes's review feedback on PR #1511: 1. Per-pubkey rate limit on POST /transcribe/session (5/min, configurable via BUZZ_TRANSCRIBE_SESSIONS_PER_MINUTE). Each session opens a metered OpenAI Realtime connection on the operator's bill. 2. Auto-submit disabled by default (DEFAULT_AUTO_SUBMIT_PHRASE = ''). The infrastructure for configurable phrases remains in place and can be wired to a user setting later. Also: - Move BUZZ_TRANSCRIPTION_MODEL into Config (consistent with other knobs) - Use a shared reqwest::Client via OnceLock (connection pooling) - Stop dictation on manual send (stopDictationRef wiring)
|
Addressing Wes's review (3 blocking + non-blocking): 1. Rate limit on
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8908e790e7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…penai_client - Stop active dictation run before the edit branch clears/saves, matching the normal-send path. Prevents late transcript events from writing back into the restored or fresh draft after an edit save. - Replace .expect() in openai_client() with proper error propagation via Result, avoiding a panic if TLS backend initialization ever fails. Addresses review feedback on PR #1511.
|
Addressed the two open review threads in cf21dcc:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cf21dcc068
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 237c29e0e4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d6bfcbe3f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5f1f5d43c4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6e8d9c4f69
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e06c632eb2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9adae3e927
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 849f8a971e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
When BUZZ_TRANSCRIPTION_MODEL is a realtime-whisper variant, the relay now omits server_vad from the session config. Without server VAD, OpenAI buffers audio indefinitely until a manual input_audio_buffer.commit is sent. This commit adds the client-side counterpart: 1. Export commitAudioBuffer() and requiresManualCommit() from realtimeAudio.ts. 2. On session creation, track whether the model requires manual commit. 3. After flushing the pre-connection buffer, commit immediately and start a 2s periodic commit interval so streaming transcripts flow during recording. 4. On stop, send a final commit before teardown and keep the data channel open briefly (3s) to receive the last transcript response. 5. Clear the commit interval on cleanup. Without this, realtime-whisper sessions would stream/buffer audio but never produce transcripts because no commit was ever sent.
Address two review comments: 1. (useRealtimeDictation.ts) Increment activeRunIdRef immediately in the manual-commit cleanup path so late transcript events from the 3s grace window are rejected by handleRealtimeEvent. Previously, the run stayed valid during the timeout, allowing transcripts to write back into the composer after a send, edit-save, or navigation. 2. (transcribe.rs) Replace as_object_mut().unwrap() with a branch that builds the correct JSON literal directly. Avoids introducing an unwrap in a production path per AGENTS.md rules.
Distinguish user-initiated stop (mic button) from send/navigation cleanup: - stopRecording(): user stop — keeps the run valid during the 3s grace window so the final commit's transcript is delivered to the composer. - cancelRecording(): send/edit-save/navigation — immediately invalidates the run so late transcripts cannot refill the cleared composer. MessageComposer's stopDictationRef now uses cancelRecording (called on send and edit-save). useComposerDictation's draftKey/disabled effects also use cancelRecording. The user-facing toggleRecording/stopRecording path preserves the final transcript for short recordings.
Add expires_after: 60 (seconds) to the OpenAI Realtime client-secret request payload. This limits the reuse window of minted secrets — a client must establish its WebRTC connection within 60s, and cannot reuse the secret to open additional sessions after that. Without this, the default 600s TTL allowed a compromised or malicious client to bypass the per-pubkey rate limiter by reusing a single minted secret for many concurrent sessions.
…dels Two fixes: 1. (transcribe.rs) expires_after must be an object with anchor + seconds fields per OpenAI's client-secret schema, not a bare integer. The bare number would cause OpenAI to reject the request with a 400, making dictation fail with a 502 from the relay. 2. (useRealtimeDictation.ts) The user-stop grace window (3s delayed teardown with run kept valid) now applies to ALL models, not just manual-commit ones. For server-VAD models, the final VAD completion event may still be in-flight when the user clicks the mic to stop — without the grace window, that event was dropped and the tail of the dictated message disappeared.
1. Proxy SDP exchange through the relay (transcribe.rs) The relay no longer returns the raw OpenAI client secret to the desktop client. Instead, POST /transcribe/session returns an opaque session ID, and the new POST /transcribe/sdp endpoint accepts the client's SDP offer, looks up the cached secret, forwards it to OpenAI, and returns the SDP answer. This prevents a compromised client from reusing the bearer token to open non-transcription Realtime sessions under the operator account. 2. Allow send during the transcribing grace window (MessageComposer.tsx) Previously, pressing Enter/Send while isTranscribing was true (during the 3s grace window after user-stop) blocked the send entirely. Now the send proceeds with whatever content is already in the composer (transcript deltas have been applied incrementally) and cancels the dictation run so late events don't refill the cleared composer.
1. Merge session + SDP into single POST /transcribe/connect The two-step flow (POST /transcribe/session → POST /transcribe/sdp) stored the OpenAI client secret in an in-memory DashMap, which breaks in HA deployments where the two requests may land on different relay replicas. Replaced with a single POST /transcribe/connect that accepts the SDP offer, mints the OpenAI session, proxies the SDP exchange, and returns the SDP answer + model — all in one request. No server-side session state is needed between requests, so this works correctly across any number of replicas. Removed the transcribe_sessions DashMap from AppState entirely. 2. Clear isTranscribing even when final text is unchanged When a TRANSCRIPT_COMPLETED_EVENT produces no text change (final matches accumulated deltas), the early-return in handleRealtimeEvent skipped the setIsTranscribing(false) call. This left the 'Transcribing…' indicator stuck indefinitely. Fixed by updating the transcribing flag before the text-change early return.
OpenAI emits conversation.item.input_audio_transcription.failed when an item cannot be transcribed (e.g. audio_unintelligible). Previously this event was dropped by the type filter, leaving isTranscribing stuck true with no user feedback after a preceding buffer commit. Now the failed event is treated as terminal: clears the transcribing indicator and surfaces the error message via toast.
…pipeline Phase 1: Extract reusable SttEngine from huddle/stt.rs into stt_engine.rs. The core STT logic (resample 48→16 kHz, earshot VAD, sherpa-onnx Parakeet inference) is now a standalone component configurable via SttEngineConfig. huddle/stt.rs becomes a thin wrapper that passes huddle-specific flags (TTS barge-in, PTT gating). drain_until_shutdown moves to stt_engine and is re-exported by huddle/mod.rs for backward compat. Phase 2: Add Tauri dictation commands (start_dictation, stop_dictation, push_dictation_audio, get_dictation_status) that create a standalone SttEngine instance with dictation-tuned settings (longer silence threshold, no TTS/PTT flags). Transcribed text is emitted to the frontend via 'dictation-transcript' Tauri events. Phase 3: Add useLocalDictation hook that captures mic audio via AudioWorklet and sends raw PCM to the native STT engine. useDictation now routes to local STT when available (offline, no API key), falling back to cloud (OpenAI Realtime via relay) when the model isn't downloaded. Key wins: - Works fully offline — no BUZZ_OPENAI_API_KEY needed - Self-hosters get dictation for free - Lower latency (no network round-trip) - No relay billing concern - Cloud fallback preserved for higher accuracy
- Remove useRealtimeDictation, realtimeAudio, realtimeBufferWorklet, and transcribeSession API — dictation is now local-only (Parakeet). - useDictation simplified to use only useLocalDictation. - Add ⌘D (Ctrl+D on Windows) global shortcut to toggle dictation. - DictationButton tooltip shows 'Voice Dictation ⌘D' with kbd styling matching the search bar's ⌘K pattern. - Shortcut registered in keyboard-shortcuts.ts settings list. - AppShell dispatches custom event; useComposerDictation listens.
- ⌘D is now push-to-talk: hold to record, release to stop. AppShell dispatches keydown/keyup events; useComposerDictation starts on keydown and stops on keyup. - Button click still toggles (click mic to start, click stop to end). - Mic icon swaps to a filled stop square while recording. - Shortcut description updated to reflect hold-to-talk behavior.
Three fixes for the push-to-talk dictation experience: 1. **Streaming partial transcripts**: The STT engine now flushes a partial transcript every ~2 seconds of continuous speech (DICTATION_PARTIAL_FLUSH_SAMPLES). Previously, text only appeared after a silence gap or when the user released the key. Now text streams into the composer while you're still speaking. 2. **Final flush on shutdown**: When the STT worker exits (key released or stop_dictation called), it flushes any remaining speech buffer before shutting down. Previously, speech accumulated since the last silence gap was silently discarded. 3. **Suppress macOS accent picker**: Holding ⌘D triggered the macOS press-and-hold accent character popup (showing ð, đ, etc.). Fixed by setting ApplePressAndHoldEnabled=false for the app's bundle ID at startup. Also fixes the event ordering: the 'dictation-state: stopped' event is now emitted by the forwarder task after all pending transcripts have been delivered, ensuring the frontend receives the final text before the stopped signal.
…e shortcut, batch audio Addresses all 8 unresolved review threads on PR #1511: 1. **Accumulate local transcript segments** (already fixed) — each streaming partial resets lastTranscriptRef so segments are appended, not replaced. 2. **Flush local speech before shutdown** (already fixed) — worker flushes remaining speech_buf on loop exit. 3. **Scope dictation shortcut to one composer** — new activeComposer module tracks which composer instance last received focus. The ⌘D handler only dispatches to the active instance, preventing duplicate recordings when both channel and thread composers are mounted. 4. **Keep listening through user-stop finalization** (already fixed) — stopRecording no longer calls cleanup(); event listeners stay alive until the native 'stopped' event arrives with the final transcript. 5. **Refresh dictation availability after model downloads** — availability check now polls every 5s until the model is ready, then stops. Covers the fresh-install case where the model downloads in the background. 6. **Stop native engine after startup failures** — catch block now calls invoke('stop_dictation') before cleanup so the native SttEngine doesn't linger when mic permission is denied or AudioWorklet setup fails. 7. **Prevent disabled composers from shortcut-starting** — keydown handler checks disabledRef/isSendBlockedRef before starting dictation. 8. **Batch dictation audio before IPC** — worklet frames are accumulated in a Float32Array batch and flushed every 100ms (~10 IPC calls/s instead of ~375). Also removed the erroneous worklet→destination connection that was playing mic audio back through speakers. Additionally: - Removed the entire dead OpenAI relay proxy (transcribe.rs, routes, config fields, .env.example entries, reqwest dep, bridge.rs visibility widening). - Scoped the keyup event to only fire when ⌘D keydown was actually dispatched (no more spurious events on normal 'd' typing).
…await flush before stop
Addresses 3 new review comments:
1. **Avoid cancelling another composer's dictation** — the draftKey effect
now checks isRecordingRef before calling cancelRecording(), so switching
channels in one composer won't kill a recording in another.
2. **Stop native dictation on unmount** — cleanup() now calls
invoke('stop_dictation') so the native SttEngine doesn't linger when
the hook unmounts mid-recording (navigation, composer close).
3. **Await final audio flush before stopping STT** — stopRecording() and
cleanup() now await the flushAudioBatch() promise before invoking
stop_dictation, ensuring the last batch of audio arrives at the native
engine before it shuts down and flushes its speech buffer.
Addresses 2 review comments: 1. **Tag dictation events by session** — each startRecording increments a session counter. Transcript and state event listeners check the session ID and ignore events from a previous session's forwarder, preventing stale transcripts from leaking into a new draft. 2. **Abort pending starts** — stopRecording and cancelRecording set an abort flag. startRecording checks this flag after each async await (listener setup, engine start, getUserMedia) and bails early if the user already stopped/cancelled, preventing zombie worklets and timers.
…ering
The native dictation forwarder now includes a monotonically increasing
session ID in every emitted event payload:
- Transcript events: { text: string, session: number }
- State events: { state: 'started'|'stopped', session: number }
start_dictation returns the session ID so the frontend stores it in
nativeSessionRef before registering listeners. Listeners compare the
event's session field against nativeSessionRef.current and ignore
mismatches, definitively preventing stale transcripts from a previous
session's forwarder from leaking into a new draft.
Also reordered the start flow: engine start (which returns the session ID)
now happens before listener registration, eliminating the timing gap where
listeners could accept events with an unknown session.
Each listener now compares event.payload.session against the local `sessionId` const (captured by the closure) instead of reading from the mutable `nativeSessionRef.current`. This prevents stale listeners from a previous session from matching the new session's ID after the ref is updated. Also unregisters any lingering listeners at the start of a new session before registering fresh ones, ensuring no duplicate handlers exist.
… stop Address two P2 review comments on useLocalDictation: - Recheck startAbortedRef after audioWorklet.addModule() resolves. If stop/cancel fired during the module load, the start path previously continued to wire up the worklet, start the flush timer, and set isRecording=true, leaving dictation running after it was stopped. Now bails and tears down the mic/context it opened. - Clear worklet.port.onmessage on stop/cleanup so PCM messages still queued on the main thread are dropped instead of appended to the reused audioBatchRef after the final flush. Also reset the batch on cleanup and at the start of each new recording so a stale batch can't leak into the next session/draft.
klopez4212
force-pushed
the
kennylopez-dictation
branch
from
b7402c3 to
1d9610b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1d9610bbde
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Address two P2 review comments: - stt_engine: the worker broke out of its loop as soon as the shutdown flag was set, skipping any audio already enqueued by the caller's final push_dictation_audio flush. The shutdown flush then only transcribed older speech and dropped the last words. Now drain and process everything still in the audio channel before the final flush. - dictation/useLocalDictation: stop_dictation now takes an optional session id and only tears down the engine when it matches the currently-stored session_id. stopRecording captures its session and passes it to the deferred (post-flush) stop, so restarting during the Transcribing grace window no longer lets the stale stop kill the newly-started session. Unconditional callers (cancel, unmount, abort-bail, start) pass None.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 12b5c5859e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The unmount cleanup handler (registered for every useLocalDictation instance) called stop_dictation unconditionally, so a non-recording composer unmounting (e.g. closing a thread reply composer) while another composer was actively dictating would tear down the singleton engine and drop the real recording's final transcript. Scope every frontend stop_dictation call to a specific session: - cleanup() passes nativeSessionRef.current (0 for an instance that never started; native session IDs start at 1, so a never-recorded composer's stop can never match the live session and correctly no-ops). - startRecording bail paths pass the just-started sessionId, robust against a concurrent restart during an await window. - error path and cancelRecording drop their redundant unconditional stop and rely on cleanup()'s scoped stop.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9d0441453
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Two P2 review findings: 1. Resume the AudioContext before wiring the worklet. The WebView can create the context in the 'suspended' state after the async start path (mic prompt etc.); a suspended context never pulls the worklet's process(), so no PCM reaches push_dictation_audio while the UI shows an active recording. Mirror the huddle capture path and resume() when suspended. 2. Chunk merged PCM under the native 100 KB IPC cap. push_dictation_audio rejects any raw batch over MAX_AUDIO_BATCH_BYTES (100 KB = ~0.53s at 48kHz f32 mono). If the batch timer is delayed (main-thread stall) the merged buffer can exceed the cap, native rejects it, and the .catch silently drops the whole chunk. Split into <=24,000-sample (96 KB) chunks sent in order.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b0dae02c5f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…io pushes Address two codex P2s on cross-session/late-arrival races: 1. Cancel dictation during the final transcribing window. The draftKey cancel effect only checked isRecording/isStarting, but stopRecording() keeps the transcript listener alive (isTranscribing=true) until the native stopped event. A channel/thread switch during that grace window didn't cancel, so a late transcript leaked into the newly restored draft. Include isTranscribing in the ownership check. 2. Scope push_dictation_audio to the owning session. Late flush chunks from a just-stopped session could be fed into a newer session's engine and transcribed into the new draft. Prepend an 8-byte LE u64 session header to each raw audio payload; native only feeds audio whose header matches the active session_id and drops stale chunks.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9937adbad8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
cleanup() tore down resources but never set startAbortedRef, so a composer unmounting while startRecording() awaited start_dictation / listen / getUserMedia / addModule would resume the async start against a torn-down instance — opening the mic/worklet and leaving the native session and event listeners running after the component was gone. Set the abort flag first in cleanup(); a fresh startRecording clears it before its first await, so subsequent starts are unaffected.
3 participants
Summary
Adds real-time voice dictation to the message composer using OpenAI's Realtime API over WebRTC.
How it works
POST /transcribe/session)Relay changes (
crates/buzz-relay)POST /transcribe/session— mints an ephemeral OpenAI Realtime client secretGET /transcribe/status— returns whether transcription is configuredBUZZ_OPENAI_API_KEYenv var — no key = mic button hidden (graceful degradation)reqwestas a direct dependency for the upstream HTTP callDesktop changes (
desktop/src/features/dictation/)lib/realtimeBufferWorklet.tslib/realtimeAudio.tslib/voiceInput.tsapi/transcribeSession.tshooks/useRealtimeDictation.tshooks/useDictation.tshooks/useComposerDictation.tsui/DictationButton.tsxIntegrated into
MessageComposervia the toolbarextraActionsslot.Configuration
Design decisions
reqwestworkspace dep.