Voice-call the session's AI agent from the share viewer (Boss Calling) - #345
Conversation
…ewer Adds a "Call" pill to the share viewer that connects the remote user by voice to an OpenAI Realtime agent able to inspect and act on the shared session. Audio goes browser↔OpenAI over WebRTC (never through the tunnel); the host mints a short-lived ephemeral client secret from the user's own key, and the agent's function calls ride the existing share WebSocket (voiceToolCall → executed against the shared session → voiceToolResult). - new compose/voice package: key storage (chmod-600 ~/.bossterm/voice.json), curated tool catalog, GUI + daemon executors, mint broker, shared service - 7 additive share-protocol messages; wired into MirrorShare and DaemonShareServer, both gating on control role and share scope - one shared "Boss Calling (voice)" section in Settings AND both share windows, so the key can be added where the link is handed out; its status line mirrors what the viewer is told, since a missing key otherwise shows as nothing at all in the browser - on by default: storing a key is the deliberate opt-in, and with no key the host advertises itself unavailable - viewer pill, mute/end controls, toasts; connect-src names api.openai.com (the SDP exchange), the only remote origin the viewer may reach
…sh model list
The GA SDP endpoint takes no URL parameters — any query string comes back as an
empty 400 — and the model is already bound to the ephemeral secret the host
mints, so passing it was both wrong and redundant. VoiceSession.model is now
purely informational: the viewer shows it on the Call pill during a call.
Also refreshes the model choices, which had gone stale: gpt-realtime-2.1
(new default), gpt-realtime-2.1-mini, gpt-realtime-2, gpt-realtime.
Verified against the current API docs: the mint body is correct as-is
(expires_after{anchor,seconds}, session.type=realtime, output_modalities,
audio.input.turn_detection semantic_vad, audio.output.voice, tools,
tool_choice), as are the data-channel event names and the oai-events channel.
Review: Boss CallingNice piece of work — the architecture is the right one. Audio stays off the tunnel, the API key never leaves the host, every tool call is re-validated host-side rather than trusting the model's scoping, the mic prompt is inside the click gesture, and the mint error path deliberately keeps OpenAI's body host-side with the request id. The Findings below, most-significant first. (Posting in two comments — it got long.) 1.
|
|
(review, part 2) 6.
|
Review — Boss Calling (1/2: the significant stuff)Really nice piece of work. The architecture is the right one, and the key decision — the host re-validates every tool call server-side instead of trusting the browser — is what makes it sound. A malicious viewer can Findings in rough severity order. 1.
|
Review — Boss Calling (2/2: smaller findings, tests, nits)6.
|
…d call Calling is a mode you're in, not a toolbar toggle, so it gets its own bar at the bottom of the viewer instead of a badge among the top-bar chips. Idle it is a single Call button. On a call it becomes a strip: a level meter, what the agent is doing right now (Connecting… / Listening… / Agent speaking / Working… / Muted), Mute, and End call. The meter is real audio, not a canned animation — an AnalyserNode on the mic while you talk and on the agent's own track while it answers — so "it can't hear me" is visible at a glance. Without WebAudio the bars stay flat rather than breaking the call. The bar stacks above #keybar and both reserve space via #body's padding, so the on-screen keys, the call controls and the terminal never overlap. Harness: teach viewer-harness.cjs to drive a call (secure context, mic, WebRTC, fetch) and add two scenarios — the full Call → live → tool → End flow, and a socket drop mid-call. The runner now awaits scenarios so a promise-driven flow can't lose its assertions to an unhandled rejection. This also pins the SDP endpoint against the ?model= regression.
|
Review: Boss Calling (voice-call the session AI agent) — part 1/3 Read the whole diff. Well-structured feature: the Findings most significant first. Nothing challenges the design, but items 1-4 are worth fixing before merge — they all sit inside the security envelope the PR explicitly claims. 1.
Fix: re-check 2. Every other path validates the target (GuiVoiceToolExecutor.kt:80-83), but // GuiVoiceToolExecutor.kt:77
"get_active_tab" -> return tabInfoJson(defaultTabId ?: anchorTabId)
Exploitability is limited: ids are random UUIDs (TerminalTab.kt:49, TerminalSessionCore.kt:57), so they cannot be guessed. But a viewer that legitimately learned ids while a share was ALL-scoped keeps working after the scope narrows, and this is a one-line asymmetry against a boundary enforced everywhere else. Better than patching 3. No rate limit on
|
|
Review — part 2/3 4. Disabling Boss Calling mid-call leaves the call running with no way to hang up
var available = !!(voice.status && voice.status.available);
voiceBarEl.classList.toggle("on", available);
if (!available) { voiceCallEl.classList.remove("on"); layoutForKeyboard(); return; }End call lives inside that bar. So if the host toggles Boss Calling off during a call, the viewer loses the hangup control while the mic stays hot, the peer connection stays up, and (per item 1) the tool bridge keeps executing. Add 5. One case "voiceToolResult":
voiceDcSend({ type: "conversation.item.create", item: {...} });
voiceDcSend({ type: "response.create" });
6. BossTermMcpServer(config = BossTermMcpConfig(allowWriteTools = true)).createServer()
7. The class doc says 8. runCatching { ... }.onFailure { log.warn("Could not persist voice config: {}", it.message) }
_keyPresentFlow.value = config.openaiApiKey.isNotBlank()On a failed write (read-only home, full disk, |
|
Review — part 3/3 9. Test coverage Solid on
10. Smaller notes
I did not run a build or the test suite as part of this review, so the green claim is taken at face value. Agreeing with the PR own caveat too: with no end-to-end call yet, the mint response shape ( |
… "Call BossTerm" The key row now matches every other settings action row (SettingsFilePicker): one accent-filled 36.dp primary plus a quiet TextButton, instead of a Danger-tinted filled block that read as foreign next to the Share window's own buttons. Clear key only appears when there is a key to clear. The viewer's button says "Call BossTerm" rather than a bare "Call", and the settings copy + availability line name it the same way. The harness pins the label so it can't drift.
Review — Boss CallingRead through the whole branch. The architecture is good: A handful of things I'd want fixed before merge, roughly in severity order. 1.
|
…witch a kill switch Review fixes, most consequential first. 1. get_active_tab / list_tabs escaped the share scope. Both executors dispatched the locally-answered tools before any scope check, and their lookups are global (McpTerminalRegistry.findTab spans every window; host.list() every session). defaultTabId is unvalidated viewer input, so a viewer on a SESSION-scoped share could Focus a foreign tab and read back its title and cwd. The check now precedes every dispatch, and Focus/VoiceStart refuse to store an id the share doesn't cover in the first place. 2. response.create fired mid-response and once per parallel call, which earns conversation_already_has_active_response. The viewer now buffers function_call_output and asks for exactly one follow-up turn, after response.done AND once every call of that round has resolved. Speaking and tool-busy became separate flags, so audio starting during a tool no longer desyncs the bar. 3. The master switch didn't stop a call already running. handleToolCall re-reads it and invalidates live calls; the viewer ends the call when voiceStatus goes unavailable instead of just hiding the bar. 4. VoiceAgentStorage.save flipped the UI to "key is set" even when the write failed — which also made shares advertise available:true while load() returned null. It now returns success, only publishes on success, cleans up the staged temp file, and the settings UI surfaces the failure. Writes moved off the Compose thread. 5. Nothing bounded mints or bound tool calls to a call. Added a mint gap + rolling window cap (each mint is billed against the host's key), and a call token minted in voiceSession and echoed on every voiceToolCall, so the share socket can't be driven as a standing tool RPC. 6. GuiVoiceToolExecutor fabricated its own BossTermMcpConfig, so an embedder's read-only surface still exposed run_command by voice and a toolNamePrefix would have collapsed the surface. The manager now publishes the real config on McpTerminalRegistry and the executor honours it, prefix included. 7. A call is remote hands on the machine with nothing visible host-side, and the feature ships enabled — call start/end now notify through the same channel as a share approval request. Also: atomic in-flight claim (get-then-increment could exceed the cap), status() answers from the in-process key flow instead of re-reading voice.json per settings emission, daemon shares poll voice availability so a key added mid-session reaches connected viewers, dropped the dead VoiceSession.expiresAtMs, cached the meter's bars instead of re-querying at 60Hz, and imported JsonPrimitive rather than fully qualifying it three times. Tests: scope-escape regression (get_active_tab pointed out of scope, as claimed current tab and named outright), no-token/forged-token refusal, kill switch mid-call, mint rate limiting, and the harness now asserts the token echo, the one-response.create-per-round contract, and revoke-ends-the-call.
Settings → Session Sharing is where someone looks to find out why the viewer shows no Call BossTerm button, so it should answer that in place rather than only in the Share window.
Review: Boss Calling — correctness & security (1/2)Read the whole diff (27 files, ~2.8k lines). Unusually well-constructed for a feature this size: policy centralised in one (Reviewed by reading; I did not run a build — the PR states 1.
|
Review: Boss Calling (PR #345)Read the whole diff — the seven new 1. No watchdog on the tool round-trip — a call can wedge silentlyThere's a 15 s watchdog for
Suggest a per- Smaller edge in the same function: 2.
|
Review: Boss Calling — performance, tests, nits (2/2)Performance
Test coverageStrong: the loopback mint stub, the 5xx-doesn't-leak-account-detail test, the exact-CSP assertion, and driving the real
Nits
Things I checked and found correctRecording these so they don't get re-litigated:
Nice work on the security posture in particular: scope-checking before dispatch including the locally-answered tools (with a test naming exactly why that matters) and keeping the key out of |
Adds a "Call BossTerm" pill to the host window's status strip, beside Sharing, and an in-app call bar: talk to this terminal's agent from the app itself. No new dependency and no WebRTC. The Realtime API also speaks plain WebSocket, so the host connects over the JDK's own java.net.http WebSocket with its standard API key (there's no untrusted party here, so no ephemeral secret), and javax.sound.sampled handles the PCM16 24kHz mono capture and playback the API expects. - RealtimeTransport: the socket, behind an interface so the protocol is testable - VoiceAudioIo: mic + speaker on dedicated threads, NOT a shared dispatcher — a capture loop blocks for the whole call, which is how BossTerm starved its IO dispatcher once before - HostVoiceCallController: session.update handshake, mic streaming, playback, barge-in (drop queued agent audio when the user starts talking), and the same one-response.create-per-tool-round rule the browser path needed - Tools run through the existing VoiceToolExecutor, so both surfaces expose one curated set; scope is this host's own tabs, defaulting to the focused one - UI: status-strip segment (idle / calling / live / speaking / working / failed) plus a bar with a real mic level meter, Mute and End call - macOS packaging: NSMicrophoneUsageDescription + the audio-input entitlement in both entitlement files — without them a packaged app is denied the mic GuiVoiceToolExecutor's anchor tab became a provider: a share anchors on its own tab, but an in-app call follows whichever tab is focused. Tests drive the machine with a fake socket and fake audio (CI has no mixer): session/audio config, base64 mic appends, mute stopping the stream, playback and barge-in, a two-call round emitting exactly one follow-up turn, call-id dedupe, no-key/disabled refusing before any socket opens, and teardown releasing both.
The API rejects the session with "Missing required parameter:
'session.audio.output.format.rate'" — the guide's WebSocket example shows the
output format as {type: audio/pcm} with no rate, but it is required on both
directions. Pinned in the test so it can't regress.
Review: Boss Calling (#345)Read the whole diff. This is high-quality work — the security reasoning is unusually explicit (ephemeral-secret-only to the browser, key in a separate chmod-600 file, server-side Below: one bug I'm fairly confident is real, a few correctness/perf issues, then smaller notes. 1. The daemon's voice-status poller almost certainly never runs (bug)
For the first viewer of a share, Fix: move Also 2.
|
|
Review: Boss Calling Read the whole branch — the voice package, both share-server integrations, viewer.js, settings/packaging, and the new tests. This is unusually well-constructed for its size: the security model (host-only key, ephemeral Scope note: the description covers only the share-viewer path, but the branch also ships a complete in-app "Call BossTerm" feature ( 1. watchVoiceStatus almost certainly exits immediately for the first viewer
2. JdkRealtimeTransport.send is not safe for the traffic it carries
Same file: 3. Playback blocks the socket callback thread, defeating barge-in
4. HostCallState updates race, and recompose at audio rate
Perf: 5. Call tokens outlive the call by up to 6 hours
6. Double-clicking Call tears down the call it just made
7. Test coverage: the GUI executor is the untested one
Smaller notes
Worth keeping as-is
|
… tokens
Feature: clicking "Call BossTerm" with no key now opens a dialog to add one and
places the call on save, instead of erroring. The pill therefore shows whenever
the feature is on (new NeedsKey state) rather than hiding until the user finds
Settings.
Review fixes:
1. The daemon voice-status poller never ran. watchVoiceStatus() was armed during
the admit preamble, ~200 lines before def.viewers.add(vc), and its guard is
`viewers.isNotEmpty()` — so for a share's first (usually only) viewer the loop
exited immediately and cleared its own handle. Now armed after registration.
Its check-then-assign guard is also under a lock, so two concurrent admits
can't launch two pollers with one nulling the other's handle.
2. voiceEnd was a no-op, so tokens outlived their calls by up to the 6h TTL — a
viewer that hung up (or was later demoted to view-only) kept a working handle
for the read tools, which reach further back than the mirrored screen. Added
closeCall(token); VoiceEnd and viewer disconnect both retire the connection's
token in either server, which also makes the "call ended" notification fire
in normal operation instead of only when the master switch flips.
3. HostCallState was read-modify-written from four threads (25 Hz level ticks,
the socket thread, tool coroutines, the UI). All eight sites use update {}.
4. The mic level no longer lives in the call state: it moved to its own flow, so
a 25 Hz stream can't recompose the lambda that owns terminal rendering, which
now subscribes to a derived distinctUntilChanged segment instead.
5. connect() no longer parks a dispatcher thread on .join() (moved to IO), and
the model is URL-encoded — settings.json is user-editable, and a stray
character threw URISyntaxException out of connect instead of failing the call.
Smaller: playback drains on its own thread through a bounded queue, so a stalled
speaker can't stall event dispatch and break barge-in exactly when audio is
backed up; HostVoiceCall cancels its mirror job instead of leaking a collector
per call; start() checks the switch before the key, matching status(); a stray
voiceToolResult no longer banks a spurious response.create; VoiceAgentStorage
probes the key off-thread rather than reading disk at class-init; the
availability line no longer claims only controllers see the button; imports
replace inline fully-qualified names.
Not changed: HostCallBar and StatusStrip are both androidx.compose.material3 —
they already match. BossCallingSection is M2 because SettingsComponents is.
…er the GUI executor Closes the three findings from the latest PR review that the earlier round missed. 1. WebSocket.sendText was fire-and-forget with two concurrent producers (the capture thread at ~25 mic frames/sec and tool coroutines). The JDK contract is to wait on the returned future, so collisions threw IllegalStateException into a runCatching that logged only a class name — silently dropped mic audio, presenting as "the agent can't hear me". Sends now go through a single-consumer queue whose writer joins each future. Also: one shared HttpClient instead of one per connect (never closed before Java 21, so each call leaked a selector thread), and close() lets the close frame flush instead of aborting over it. 2. Two quick clicks on Call made two mic streams and two voiceStarts, and the second's rate_limited error tore down the call the first had established. The click now claims the call synchronously before awaiting the mic and reverts on denial; a stray stream is stopped rather than orphaned (the mic indicator used to stay lit); rate_limited no longer ends a live call and has real copy. 3. GuiVoiceToolExecutor had no tests despite being the half that carries run_command and enforces the scope gate. Seven added, centred on refusing an out-of-scope tab however it arrives, plus read-only/prefixed embedder configs. Also covered: the in-flight cap admitting exactly four, and token TTL expiry. Smaller notes from the same review: list_tabs no longer errors when nothing is focused (the target is resolved lazily, named targets still scope-checked); the daemon's poller compares mtime+size instead of decoding voice.json every tick; realtime errors are classified by code/param rather than by substring-matching prose, and non-fatal ones surface in the call bar instead of only the log.
Review: Boss Calling — part 1 of 2 (what works, and the two issues worth fixing before the first real call)I read the whole diff: both share servers, This is well built. Things that stand out as genuinely good rather than merely present:
Your own caveat — no verified end-to-end call — is the real merge gate, and I'd keep it that way: the mint body, 1.
|
Review: Boss Calling — part 2 of 23. A stale
|
Review: Boss Calling (PR #345)Read through the whole diff. This is genuinely high-quality work — the KDoc explains why rather than what, the concurrency reasoning is careful ( Below: things I think are real bugs, then design points, then nits (second comment). I did not build or run anything. 1. The minted call token is never attached to the connection, so hang-up/disconnect cleanup silently no-ops
Related, same lines: Suggested shape — capture on the way out, and only ever trust a token the host issued: is ClientMessage.VoiceStart -> {
rememberVoiceTab(vc, msg.activeTabId)
voiceService.handleStart(msg, vc.canControl) { m ->
if (m is ServerMessage.VoiceSession) vc.voiceCallToken = m.callToken
vc.outbox.trySend(ShareProtocol.encodeServer(m))
}
return
}and drop the assignment in the 2.
|
3.
|
…e-focus fallback Closes the findings from the review of 24eed2a that weren't already covered. 1. A voiceToolResult could be dropped on the GUI path and wedge the call. MirrorShare replied through BoundedViewerOutbox.trySend, which evicts the OLDEST queued frames under back-pressure — and the moment that matters is a run_command flooding output, when the queued result is the oldest thing there. BoundedViewerOutbox now has a control lane that is never evicted and drains first (the counterpart of FrameOutbox.sendControl on the daemon side), and voice replies use it. 2. Viewer side of the same: a per-call timeout now feeds an {"error": …} function_call_output back so a lost reply can't pin the bar at "Working…" with the agent mute and no way out but End call. run_command gets a long leash (its own clamp is 600s); reads time out in 45s. 3. A stale defaultTabId broke the entire tool surface. The focused tab is validated when stored but can close afterwards, and treating that as a scope violation failed even list_tabs/get_active_tab — which take no tab_id and left the agent no way to discover a live id. A tab the model NAMES is still refused (that is the boundary); stale focus falls back to the anchor, which is in scope by construction. Both executors. Smaller: the Call BossTerm indicator gets its own voiceShowStatusIndicator toggle, so it doesn't appear for users who turned the other two indicators off; daemon shares now send the same host-side call notification (the case where nobody is watching a window); stopping a share mid-call retires its calls; the playback thread sets `running` itself, so a play() before startCapture can't leave the agent silently inaudible; pollVoiceStatus compares job identity instead of isActive (which is always true inside finally, so the old guard never fired); at capacity a new call is refused rather than silently evicting a live one; TabbedTerminal imports instead of a dozen fully-qualified names; unused imports dropped; the toggle copy now says terminal contents are sent to OpenAI. Two earlier tests asserted the pre-fix strict behaviour and were updated: named out-of-scope tabs are still refused, stale focus falls back — and either way the foreign tab's data must never come back, which is what they now assert.
…live tool disables Closes the review of bd7e4e6. 1. The call token never reached the connection. It was captured only from an INBOUND voiceToolCall, so a call where the agent used no tool left nothing to retire — the token stayed valid for its 6h TTL after hangup (read_scrollback reaches further back than the mirrored screen), and the host never got the "call ended" notification that pairs with the security-relevant "call started". Worse, the inbound value was unvalidated: one bogus token clobbered the tracked one and defeated disconnect cleanup entirely. Both servers now capture the token from the host's own VoiceSession reply and never trust one off the wire. 2. Ending during "Connecting…" could leave the microphone open for the life of the process. The connect coroutine was unstored, so after teardown it resumed, opened the mic, and set phase = Live — with the owner reference already dropped, nothing could stop that capture again. The job is now stored and cancelled by end()/fail(), every suspension point re-checks the phase, and a capture that started anyway is stopped. CancellationException is rethrown rather than reported as "Couldn't reach OpenAI". 3. GuiVoiceToolExecutor froze disabledMcpTools at the first call, so disabling run_command mid-session (which manage_tools exists to do) left it callable by voice until restart. The private server now has the current disabled set applied on each use. 4. Args wider than the advertised schema reached the MCP handlers — run_command's real schema also takes panel/working_dir, and panel:"new_tab" would run in a tab the share doesn't mirror. Args are filtered to the keys the voice catalog declares. Smaller: the outgoing socket queue is bounded and drops oldest (it was unbounded while the writer blocks per frame), with a timeout on each send so a wedged socket can't park the writer forever; the daemon's list_tabs answers with no resolvable target, like the GUI's; an in-flight slot no longer leaks if the scope is already cancelled; the mint budget is process-wide (one key, one ceiling) and injectable so tests don't rate-limit each other; a missing speaker is cached instead of re-probed per audio chunk; the daemon's key-stamp cache re-reads periodically, since a same-length replacement inside one second is invisible to mtime+size. New test: ending while connect is still suspended must never open the mic.
Review — Boss Calling (#345) — part 1/2Read the full diff. This is a genuinely well-built feature: the key never leaving the host, the ephemeral-secret split, scope-limiting tool targets to the share's tabs, sanitizing mint errors before they reach a remote guest, and the Five things I'd want addressed before merge; smaller notes and the testing/docs comments follow in part 2. 1. Live-call slots leak — the host never records the token it minted
So any call that ends before the agent's first tool call never retires its token: the caller hangs up ( Same root cause: voiceService.handleStart(msg, vc.canControl) { m ->
if (m is ServerMessage.VoiceSession) vc.voiceCallToken = m.callToken
vc.outbox.sendControl(ShareProtocol.encodeServer(m))
}
2.
|
Review — Boss CallingI read the voice package (~4.3k lines), both servers' wiring, the voice half of Overall: unusually careful work for a feature this size. The things that normally go wrong in a realtime-audio + tool-calling bridge are addressed deliberately — single-consumer socket writer with a generation guard, two-lane outbound queue so a 1. The deadline ladder can't interrupt a CPU-bound handler —
|
Review — Boss Calling (voice)I read the new Below: two things I would resolve before merge, then substantive comments, then nits. Worth deciding before merge1. 2. No host-side gate on the first write tool. Substantive3. 4. The tool-round state machine is implemented twice, and only its constants are pinned. 5. 6. Comment-to-code ratio, and changelog comments specifically. This is my main maintainability concern. Large stretches read as a narrative of bugs fixed during development rather than a description of the code —
Suggest keeping the invariant and the why, and moving the "it used to do X" archaeology to commit messages — git already holds it, and the commit is where it is discoverable without being load-bearing. 7. iOS remote-audio playback. 8. Minor UX gap on the control upgrade. On Nits
Performance / security notes (no action needed)
I did not build or run this (per AGENTS.md), so the above is from reading only. |
Reported from a real call: the agent reacts to too much background noise. The cause is that both surfaces shipped OpenAI's `semantic_vad` with no options, and semantic VAD deliberately exposes no activation threshold — it is a model deciding whether you have finished a thought. That is genuinely better at not cutting you off mid-sentence, and it offers nothing at all for a room with a fan in it. There was no knob to turn. `server_vad` is the mode with `threshold` (0.0-1.0) plus `silence_duration_ms`, so the tunable levels use it: Noisy room threshold 0.80, 900ms silence Normal threshold 0.60, 700ms <- default Quiet room threshold 0.35, 400ms Automatic semantic_vad, unchanged behaviour The default is now "Normal" rather than Automatic, and 0.60 sits slightly above OpenAI's own 0.5 default — deliberately, since the reported problem was over-triggering. Automatic is still one click away for anyone who prefers the model's judgement about turn-ends in a quiet room. One definition for both surfaces (VoiceTurnDetection), because the mint body and session.update are exactly the pair that has drifted before; both existing tests that pinned "semantic_vad" now assert the two carry the SAME object. 698 tests (+6). Note the thresholds themselves are still unverified against a live call — they are reasoned from the API's documented ranges, so they may want tuning once you have tried them.
Review — Boss Calling (PR #345)Read the whole diff (68 files, ~11.4k added), with most attention on Short version: this is unusually careful work. The two-surface split is honest about being two surfaces, the security boundaries are enforced in one place and unit-tested there, and the browser half is actually tested — which almost never happens. The things I'd want changed before merge are mostly policy, not code. 1. The remote surface still has no host-side gate on the first write toolThis is the one I'd push on.
Agreed on both halves — and the mechanism is still missing. Meanwhile Two concrete asks:
2.
|
… slot
Review round 39 (three queued reviews).
The in-flight cap trusted the handler to finish. withTimeout and job
cancellation are COOPERATIVE — they land at a suspension point, and an MCP
handler doing non-suspending CPU work has none. search_output hands a
model-supplied regex to Regex() and scans per line, so a nested quantifier
against a long line backtracks effectively forever: the viewer's backstop
answers the model, the host logs "releasing the slot", and the `finally` that
actually releases it never runs. Four of those and every later voice tool on the
share answers "Too many tool calls in flight" for the life of the process —
precisely the outage the cap was added to prevent, caused by the cap.
Reachable by a remote viewer of a GUI share, and by the in-app agent under
prompt injection.
Two layers:
- Capacity no longer depends on the handler returning. Both surfaces reclaim
the slot on a deadline of their own; whoever gets there first wins a CAS, so
it is released exactly once. The thread stays pinned — nothing outside can
fix that — but the share keeps working.
- The accidental case is bounded at its source: search_output checks a
wall-clock budget per line AND inside the match loop, clips lines past 8k
chars (a 200k line is a progress bar or base64, and exactly the input that
makes backtracking explode), and tells the agent WHY it stopped so it
rewrites the pattern instead of retrying it.
Every other timeout test used a cancellable delay, which is why this was
invisible. The new one wedges a handler on a latch that is never released and
asserts the slot comes back anyway. The share path's tool budget is now injected
for it, matching HostVoiceCallController.toolTimeoutMs and for the same reason.
699 tests. Verified against the unfixed code — and my first attempt at that
verification passed only because I had mangled the file into a compile error and
was reading stale results.
Review — Boss Calling (voice-call the session's AI agent)I read the whole diff: the two Overall: this is unusually careful work. The security posture is coherent and the layering is right — one So most of my feedback is about policy defaults and merge readiness, not implementation. Blocking before merge1. No verified end-to-end call. You flag this yourself, and it's the single biggest risk. 2. 3. Bugs4. 5. The slot-reclaim watchdogs are never cancelled on success — 6. Blocking file IO on Design / operability notes7. The process-wide mint budget is a cross-share denial, not just a spend cap. 8. Test coverageStrong — ~180 new tests, and the ones that matter are the ones I'd have asked for: the call-token gate against forged/retired/expired tokens, the in-flight cap admitting exactly four, the 5xx never forwarding OpenAI's error text, scope rejection for a named foreign tab across every tool, POSIX perms on the key file, and the ladder invariant in Two notes:
StyleOne honest maintainability observation. The comment-to-code ratio here is very high, and a large share of it is archaeology — "this used to be X, which was a bug, here's what it did." The invariant comments are excellent and should stay — "peek the budget before SummaryNothing here is structurally wrong, and I'd be comfortable with the implementation. My merge conditions are the two policy defaults (#2, #3 — or the host-side write approval that subsumes both) and one verified live call (#1). #4 and #5 are quick fixes worth folding in on the way. |
Review round 40, plus the remainder of three queued reviews. The reclaim timers I added an hour ago were never cancelled on success. The CAS makes a late fire a no-op, so nothing was wrong — but a run_command parked a coroutine for its full 615s budget every time, and a call that runs twenty commands kept twenty of them alive for ten minutes each. Cancelled in invokeOnCompletion on both surfaces. viewer.js kept voice.seenCalls / pending / timers as plain objects, so a call_id of "constructor", "toString" or "__proto__" reads as truthy on its FIRST sighting: the call is dropped and the round hangs until the watchdog. Realtime ids are call_… today, so this is defence rather than a live bug — and the Kotlin mirror uses a HashSet and never had it, which is exactly the asymmetry the contract test exists to catch. Object.create(null) everywhere, and pinned. Also: - MirrorShare's revocation ticker ran for EVERY share whether the feature was on or not, and did its voice.json read on Dispatchers.Default — while VoiceAgentStorage spawns a probe thread specifically to keep that read off a shared pool and BossCallingSection wraps the same call in Dispatchers.IO. Same call, three opinions; now two, and it does not tick at all when voice is off. - msg.name was the one remote string echoed into a control frame unbounded. Not a memory hole (sendControl's ceiling closes the connection) — but the caller then sees a disconnect where a tool error belongs. - fail() read audio/transport/executor when its coroutine got a slot rather than when it was called, so a redial in between would have torn down the NEW call. - VoiceStatus.reason's KDoc still listed two of the four codes it emits. - The remote toggle's description now LEADS with the fact a host most needs — billed to your key, and you cannot end it — instead of burying it fourth. 700 tests, 21 viewer scenarios. The prototype-safety test failed first on the graphics state's unrelated `pending`; scoping it to `voice.` was the fix, since a test that names the wrong file is worse than no test.
Review — Boss CallingI read the full diff (66 files, ~11.6k added) rather than sampling, with most attention on Overall: unusually careful work for a feature this size. The things that normally go wrong in a realtime/voice integration are handled deliberately: the credential never leaves the host, the tool surface is a curated allow-list rather than "expose MCP", Policy, before merge
The daemon's "smaller surface" isn't smaller in capability. Instructions aren't a control against a hostile viewer. The viewer holds an ephemeral secret and its own data channel — worth confirming whether a client can Spend, as a number. Correctness — small, none blocking
PerformanceNothing I'd change. Level meter off the state snapshot, viewer meter throttled to ~25 Hz, Maintainability
TestsStrong (~190), and on the parts that matter: mint failures never leaking OpenAI's body, the token gate (forged/retired/expired), the in-flight cap admitting exactly four and refusing rather than evicting, the capture loop's zero-vs-negative read distinction, the transport lifecycle through an injected socket, POSIX perms on the key file. Injecting
Nice work. Merge preconditions I'd set: default |
Reported from a live call: "it keeps on looping to itself". There is no acoustic echo cancellation on the in-app surface. The browser gets AEC free from WebRTC's audio pipeline; javax.sound.sampled hands us a raw capture line and nothing else. So on speakers the microphone hears the agent's own voice, ships it back, and the model answers itself — a self-sustaining loop that no VAD threshold can break, because the input genuinely IS speech. That also explains the earlier "too sensitive to noise" report, which I treated as a VAD tuning problem. The sensitivity setting is still worth having, but it was the wrong lever for this. Half-duplex is the standard answer when you have no AEC: don't listen while you talk. Mic frames are withheld while the agent is speaking, while decoded audio is still queued or in the line's buffer, and for a 400ms tail after that — room reverb outlives the playback. Setting is ON by default, because speakers are the common case and the loop makes the feature unusable; turn it OFF for headphones, where there is no echo path and full duplex means you can interrupt by talking. The viewer now asks for echoCancellation / noiseSuppression / autoGainControl explicitly rather than relying on the browser default being what we want. Also: the transport logged only `javaClass.simpleName` when the event handler threw, so a real NoClassDefFoundError in a user's session came through as "Voice event handler failed: NoClassDefFoundError" with no way to tell WHICH class. Host logs get the class, the message and the throwable — the redaction rule was always about what reaches a remote viewer. 702 tests. The echo test was checked against the ungated send and fails there.
Reported from a live call: the agent cuts off mid-sentence and the audio sounds merged. It was dropping its own speech, and I caused it two days of commits ago. The Realtime API streams a response's audio as fast as it generates it; the speaker drains in real time. A thirty-second answer therefore arrives in a couple of seconds and waits in the play queue. PLAY_QUEUE_BYTES was four seconds, so every reply longer than that had its middle discarded chunk by chunk to make room for what followed — the caller hears the start, then a jump, with unrelated audio spliced together. That is exactly "cutting mid-sentence" and "voice merging". The bound it replaced was 100 ENTRIES, described as "~4s of queued speech". The description was wrong — deltas are server-sized, not 40ms frames — and I fixed the description by making the number true, without asking whether four seconds was the right number. It was not: the producer is faster than the consumer by design. The old bound was accidentally generous, which is why it never bit. Sixty seconds now (~2.9 MB), which exists only so a stalled line cannot grow without limit, and play() logs once per call if it is ever reached — a chopped reply should not be silent. MAX_DRAIN_WAIT_MS moves 8s -> 65s to match, or a long reply cleared "Speaking" while the agent was still audibly talking. My round-34 test asserted the four-second bound, so it encoded the bug rather than catching it. It now asserts the two things that matter: bounded, and generous enough that a real reply survives. Both tests fail at the old value. 703 tests.
Review — Boss Calling (1/2)Read the whole branch. This has clearly been through several rounds already, and it shows: the reasoning that used to live in review threads is now in the code, the invariants that mattered (owner-filtered The architecture is the right one and I am not going to re-litigate it. Below is what I think is still outstanding, roughly in the order I would want it settled. 1. Ship
|
Review — Boss Calling (2/2: smaller findings, coverage)5.
|
Reported with a screenshot: asking the agent to run a command still opened a bottom split instead of using the focused pane. idleFocusedPaneId keyed "is this pane free" off an EMPTY BLOCK LIST — `if (blocks.isEmpty()) return null`, commented "no evidence shell integration is reporting". But a terminal you just opened has announced a prompt and run nothing, so it has no blocks either. The commonest sequence there is — open a terminal, start a call, ask for a command — therefore always fell back to the scratch split, which is the exact behaviour the feature was built to stop. I conflated "nothing has run here" with "nothing is reporting here". CommandBlockTracker now exposes hasSeenPrompt (set on OSC 133;A), which is the actual question: has this pane ever told us anything, so that "busy" is answerable at all. A shell with no integration still falls back, because there busy and idle really are indistinguishable. The decision is now a pure shouldUseFocusedPane(...) with its own test covering all five branches. It was buried in a registry lookup and unreachable in a test, which several reviews flagged as a gap and which is how this shipped inverted for the one case that matters most. 708 tests.
Review: Boss Calling (voice-call the session's AI agent)I read the whole branch — both call surfaces, both share servers, the viewer, the tool catalog and executors, the audio/transport layers, and the settings/UI plumbing. Overall this is unusually careful work: the layering ( Worth crediting before the criticism, because these are the parts that are easy to get wrong and were got right:
Findings below, roughly in the order I would want them addressed. 1. Uninterruptible regex can pin
|
|
(review, part 2 of 2) 3.
|
The curated catalog advertised nine tools; BossTermMcpServer registers thirteen. run_in_panel, show_image and read_debug_console were excluded on the reasoning that a voice agent doesn't need them — which is a fair call for a REMOTE caller and the wrong one for the machine's owner, who already reaches all of them from their own MCP endpoint. In-app calls now advertise every registered tool, with the description and input schema taken from the server rather than restated. That means a tool added to BossTermMcpServer reaches the voice agent with no second edit, and the two can never describe the same tool differently — the drift this branch has had to fix repeatedly elsewhere. Write classification comes from the server's own read/write split rather than being guessed at the voice layer. Shares are deliberately unchanged and keep the curated catalog. That is not caution for its own sake: the declaredParameters allowlist is what stops a guest passing `panel: "new_tab"` and running commands in a tab the share does not mirror. The same allowlist code covers both kinds without a branch, because it reads whatever the def declares — the curated set for a catalog tool, the server's schema for an MCP one. Settings → Boss Calling → "Give the agent every MCP tool", on by default. 712 tests.
Review: Boss Calling (#345)Read through the whole
Findings below, most important first. 1.
|
| tool | Kotlin | viewer.js |
|---|---|---|
run_command |
Running a command… |
Working… |
search_output |
Searching… |
Working… |
send_input |
Typing… |
Working… |
send_signal |
Sending a signal… |
Working… |
CAPTION_FIXTURES in VoiceCrossLanguageContractTest only pins cases with the arg present, so the contract test passes. Unreachable in practice — all four params are required in the catalog — but both files claim to mirror each other, and the whole point of that test class is catching exactly this shape of drift. Either add the arg-less fixtures or make the viewer fall through to the same strings.
6. DaemonVoiceToolExecutor.get_active_tab still always reports isActive: true — minor
DaemonVoiceToolExecutor.kt:99-102 calls tabInfoJson(target, viewing = target), so s.id == viewing at line 145 is tautological. The comment at :135-136 says it "used to be hardcoded true"; on this path it effectively still is. The GUI executor's tabInfoJson does a real check against registry.findState(tabId)?.activeTabId, so the two surfaces answer differently for the same tool. Either give the daemon a real notion of "active" or drop the field there rather than assert something meaningless.
7. No inbound WebSocket frame cap — hardening, pre-existing
SessionShareManager.kt:772 and DaemonShareServer.kt:559 both install(WebSockets) with no maxFrameSize, so Ktor's default is effectively unbounded. MAX_ARGS_CHARS (32 KiB) and MAX_CALL_ID_CHARS are checked in handleToolCall after the frame has been fully buffered and deserialized into a ClientMessage. Pre-existing (Input has the same shape), so not this PR's fault — but voiceToolCall.argsJson is a new field explicitly designed to carry a large-ish payload from an untrusted viewer, and a maxFrameSize line is cheap while you are in here.
8. revocationTicks() can silence the whole voiceJob collector — nit
MirrorShare.kt:119-127 only emits while voiceCallEnabled is true, and combine needs every source to emit once before it produces anything. So with the feature off at share start, the collector at :242 never runs at all. Benign today — admit already sends voiceStatus, and there are no calls to revoke while the switch is off — but it means the collector's first emission depends on a flow that can be permanently silent, which is fragile if anything else is ever added to that block. An .onStart { emit(Unit) } (keeping the enabled check for the disk read) would make it unconditional without changing the polling behaviour.
Verification note: I reviewed by reading; I did not run a build or the test suite (per AGENTS.md I do not run the app, and the gradle invocation was not available in this environment). Findings 1, 5, 6 and 8 come from reading the code paths directly and I am confident in the mechanics; 4 is your own caveat restated with a recommendation attached.
Nice work — the security reasoning here is a notably higher standard than the feature required.
Review — Boss Calling (PR #345)I read the final state of the branch rather than the 57-commit history: the whole Overall this is unusually careful work. The failure modes that normally sink a feature like this are all handled explicitly and correctly: the ephemeral-secret boundary, the E2E gate before handing over a credential, per-connection token binding, mint refunds on every refusal path, slot reclamation independent of a wedged handler, the generation counter on the WS writer, the single-use audio latch, the "withTimeout can't cancel CPU-bound work" problem. The The findings below are mostly about defaults and one stale invariant, not about the mechanics. 1.
|
The in-app surface withheld the microphone for the whole of every agent reply, on the theory that a machine on speakers hears itself and answers its own voice. That self-loop turned out not to reproduce; what did reproduce is the cost, which is that interruption stopped working. Interruption runs off the server's `input_audio_buffer.speech_started`. That event can only arrive if the frames carrying the user's speech were actually sent, so withholding them didn't degrade barge-in — it removed it. Cutting the agent off mid-sentence is most of what makes a voice assistant usable, and it is not worth trading for a loop nobody hit. Removes the suppression, its `voiceEchoSuppression` setting and toggle, and the level-based barge-in detector written to work around it. A test pins the mic staying open while the agent speaks, verified against the old behaviour: with the withhold restored it fails.
Both blanket answers failed on real calls. Sending everything makes the agent talk to itself: there is no echo cancellation on this surface, the microphone hears the reply, and the server's VAD correctly calls it speech. Sending nothing removes interruption altogether, because barge-in runs off the server's `speech_started` and that cannot fire for frames withheld locally. So the gate has to judge level, and the previous attempt at that failed for a recorded reason: it demanded an absolute RMS of 0.28 before calling anything a voice. VoiceAudioIo's own comment puts conversational speech at 0.05-0.15, so the bar sat near shouting and nothing ever cleared it — indistinguishable from sending nothing. VoiceDuplexGate measures both sides instead of assuming either: the echo level from frames arriving while the agent talks, and the user's own level from frames the SERVER confirmed as speech during their turn. The bar sits between the two, so it adapts to a laptop speaker at 30% and a desk speaker at full. Where the echo returns louder than the user's own voice no bar can do both jobs; it prefers rejecting the loop and records why, so the log names the cause. The echo estimate carries across replies, which leaves no dead window at the start of each one. Interruptions log their measured level, bar, and both estimates, so the constants can be tuned from a real room rather than guessed again. Verified against both failure modes: with the gate bypassed the echo assertion fails, with it forced closed the barge-in assertion fails.
The gate cut the agent off mid-sentence on every reply. The cause is invisible from the gate's own code: play() QUEUES audio, so sound does not leave the speaker for some time after `speaking` becomes true. The frames captured in that window are room silence, not echo. Warm-up measured exactly them, anchored the echo estimate near zero, and the bar — a point between that estimate and the user's level — collapsed to the 0.03 absolute floor. The real echo then arrived at an ordinary 0.05, cleared the floor three frames running, and read as a voice. The mistake was letting a low echo estimate pull the bar DOWN. A quiet echo is not evidence that a quiet sound is the user; only the user's own measured level is evidence of that. So the bar is now the higher of user × 0.7 and echo × 1.6: the echo term can raise it for rooms where the reply comes back loud, and nothing can push it beneath what the user actually sounds like. With no measurement of the user at all the gate now withholds rather than deriving a bar from the echo estimate alone — that fallback was the same bug by another route. Only reachable when the agent speaks first, and it costs one un-interruptible greeting. Verified against the old bar: the latency scenario reproduces at bar=0.03 interrupting on frame 3, and a quiet room drops the bar to 0.045 against a 0.20 voice. Both now hold.
Measurements from a real call settle what two earlier attempts guessed at. The log line added for exactly this purpose reported: level=0.083 bar=0.038 echo=0.024 user=0.004 level=0.102 bar=0.038 echo=0.024 user=0.004 Two findings. The user estimate was 0.004 — silence — because observeUserSpeech averaged every frame between speech_started and speech_stopped, and server VAD only ends a turn after ~700ms of silence. All of it landed in the average, so the user term contributed nothing and echo x 1.6 set the bar by itself. Silent frames are now skipped, and the estimate uses a fast attack with a slow release so pauses cannot erase what a voice measures. The second finding is the reason no threshold on the microphone could have worked: the reply comes back at 0.06-0.10 RMS and ordinary speech measures 0.05-0.15. There is no level that divides them. The microphone cannot tell you whose voice it is. So the reference is the speaker, not the microphone. VoiceAudioIo.audiblePlaybackLevel reports how loud the audio currently LEAVING the device is, looked up against the line's own play position — the buffer is ~0.6s deep, so write time would mis-align the reference by more than the echo it predicts. Echo is that signal attenuated by the room, a factor the gate learns rather than assumes, and anything materially above it is a second voice. The bar therefore tracks the agent's envelope: in the gaps between its words the reference falls, the bar falls with it, and an interruption lands at once. The gate now also measures before it judges. Both previous versions started their estimate at zero and only updated it from frames already judged to be echo, so the opening frames of a reply were tested against nothing and cleared a floor. Ten frames of real playback are required first, once per call, since a room's acoustics do not change per reply. Where the echo returns at nearly the level it was played, no bar can admit the user and reject it; rejecting wins, and the log says SPEAKERS-TOO-LOUD rather than presenting as flaky barge-in. Verified against both mistakes: neutralise the reference and the echo interrupts at bar=0.060, reproducing the reported failure; key the reference off write time instead of the play position and it reports a chunk that is not audible yet.
send_input writes to the shell's stdin verbatim and presses nothing — its own contract puts the newline on the caller. Text sent without one sits at the prompt unexecuted, and the agent then reads a scrollback with no result and reports the command as hung or failed. That is a far more confusing outcome than a missing keystroke, and it is invisible from the tool's name. The requirement was only stated in two of the three places it applies: the missing-shell-integration fallback and the branch for surfaces without run_command. The branch that actually recommends send_input for interactive programs said nothing about it. It is now one rule, stated wherever the tool is available, covering shell commands, REPL lines, and answers to interactive prompts alike. \r\n rather than either character alone: a bare \n submits in most shells but not in every interactive program, and CRLF is what a terminal sends for Enter. Verified by removing the rule: the test fails.
Without this the agent attempts refactors as a series of sed one-liners narrated over voice, which is the worst available way to do them. Claude Code is already on the machine — the same CLI the AI menu launches and CliBinaryResolver resolves for MCP attach — and is far better suited to multi-step code work than a voice turn. The mechanics are stated rather than left to be discovered, because getting them wrong looks like the tool being broken: an interactive `claude` enters the alternate screen, which is exactly the case run_command refuses with "TUI detected". So the rule names `claude -p` for the non-interactive path, send_input + read_scrollback for a session, and says to type into a session already open in the pane instead of starting a second one. Only stated for surfaces that can actually launch it — a read-only embedder is not told about a CLI it has no way to run.
Review: Boss Calling — part 1 (findings that matter)Read the whole Four things I'd want settled before merge. 1.
|
Review: Boss Calling — part 2 (smaller items, perf, tests, style)Correctness / clarity
PerformanceNo concerns of substance, and several choices are right for this codebase: dedicated threads for capture and playback rather than dispatcher workers (with the IO-starvation precedent cited),
Testing45+ new tests, aimed at the right places: the protocol state machines driven by fakes, the token gate's four failure modes, the in-flight cap admitting exactly four, the 5xx never forwarding OpenAI's text, POSIX perms on the key file, and Gaps are the three named in part 1: prefixed + exposeAll, StyleOne broader note. Nothing here is a reason not to ship the feature. #1 is a straightforward bug, #2 is a one-line filter plus a test, #3 is a decision already on the table, #4 needs one call by hand. |
BossTerm's MCP surface could open tabs and splits (run_in_panel) but never close them, so an agent asked to tidy up had to say it couldn't — which came across as a permissions problem rather than a missing capability. `mcp__boss__tab_close` exists, but that is the terminal-tab plugin inside BossConsole, a different app; this server had no counterpart. One tool rather than two, because the caller's intent differs only by scope and both ids come from list_panes: `pane_id` present closes that split and leaves the rest of the tab open, absent closes the tab and every pane in it. It refuses a window's LAST tab. TabController.closeTab reaches onLastTabClosed() there, whose own comment reads "exit application" — so an agent tidying up would quit BossTerm, and over voice it would kill the call that asked. That guard is in the tool, not in a prompt: the worst outcome is the app vanishing, which is not something to leave to instruction-following. It is extracted as a pure function so the rule is actually tested; driving the handler needs a live window, and TabbedTerminalState.createTab returns null with no PTY available. Passing a pane_id for a tab with no splits is an error rather than a whole-tab close: list_panes reports a split-less tab as one pane whose id EQUALS the tab id, so an agent echoing that back is asking to close a split it thinks exists, and treating it as "close everything" would destroy panes on a misunderstanding. Registered as a write tool, so allowWriteTools = false embedders don't get it, and the in-app voice agent picks it up through the full-surface path while shares keep the curated catalog. The voice agent gets its own rule to name the target and confirm first, since unlike rm there is nothing to recover.
"Keep replies short. This is a voice conversation." named a preference without naming a behaviour, and changed nothing: the agent still restated the request, narrated its plan, and closed by offering further help — all around an answer that was often one word. The rule now lists what to STOP doing (restating, narrating a plan, explaining reasoning, listing next steps, asking if there's anything else) and gives one-word examples of what a reply usually looks like. It stays last in the block, because recency carries weight here. A second rule was working against it. "Say briefly what you are about to do before a slow tool call" invited a preamble before every call, and the preamble was usually longer than the result. Narration is now limited to work slow enough that silence looks broken — a build, an install — and anything quick is just done. The full-sentence exceptions are kept deliberately: an error the user has to act on, and any confirmation the destructive-command and close_panel rules require. Brevity must not eat a safety prompt.
Review: Boss Calling (voice-call the session's AI agent)I read the whole branch (72 files, ~13k lines) against Below: the policy call I'd want settled before merge, four concrete defects, and some smaller notes. 1.
|
1 participant
Boss Calling — voice-call the session's AI agent from the share viewer
Adds a Call BossTerm button to a bar along the bottom of the share viewer. Clicking it connects
the remote viewer by voice to an OpenAI Realtime agent that can read the shared terminal, run commands
in it, and narrate what it finds — so someone holding a share link can talk to the session instead
of typing into it. While a call is up that bar becomes the call strip: a live mic level meter, what the
agent is doing, Mute and End call.
Off-hours motivation: the viewer already lets a phone watch and steer a session; the missing piece
was doing that hands-free.
How it works
voiceStart.POST /v1/realtime/client_secrets, the GA endpoint)using the key stored on the host, baking in the model, voice, instructions, and tool schemas.
POST /v1/realtime/calls?model=…).Audio never rides the tunnel or the share socket.
the host executes them against the shared session and replies
voiceToolResult, which theviewer feeds back as
function_call_output.Second surface: "Call BossTerm" in the host window
The same feature also works with no browser at all. A pill in the host window's status strip (beside
Sharing) starts an in-app call: the host talks to its own terminal's agent.
Its posture is deliberately different, and worth reviewing separately:
ek_…minted per calljavax.sound.sampled, PCM16 24 kHz monoNo new dependency: the Realtime API also speaks plain WebSocket, and the JDK has had both a
WebSocket client and audio capture/playback for years. Mic and speaker each run on a dedicated
thread (a capture loop blocks for the whole call — that is how BossTerm starved its IO dispatcher
once before), playback drains through a bounded queue so a slow speaker can't stall event dispatch
and break barge-in, and clicking End during "Connecting…" cancels the connect rather than letting it
resume and open the mic behind a torn-down call.
macOS packaging gained
NSMicrophoneUsageDescriptionplus theaudio-inputentitlement in bothentitlement files — without them a packaged build is simply denied the mic.
Security
~/.bossterm/voice.json(a separatefile from
settings.json, which is world-readable and ends up in bug reports) and is used onlyas the
Authorizationheader on the mint call. The browser only ever sees a short-livedek_…secret scoped to the exact tools and instructions the host baked in.
canControl. A view-only viewer gets an explicitnot_controllerrather than silence.list_tabsisanswered from the registry and filtered, and an out-of-scope
tab_idis rejected. The agentcannot read or type into tabs the share doesn't cover.
connect-srcnameshttps://api.openai.comand nothing else new; it is the only remote originthe viewer may reach. Named unconditionally because CSP is baked at page load, so gating it on
the enable toggle would break a call started after the host flipped it.
error.message+x-request-idhost-side, while the remote callergets only the status code — a 5xx body can name the org/project.
Tool surface
Curated 1:1 onto existing BossTerm MCP tools:
list_tabs,get_active_tab,list_panes,read_scrollback,search_output,get_last_command, and the write toolsrun_command,send_input,send_signal. Deliberately excluded:run_in_panel(subsumed),show_image(renders on the host screen — useless to a remote caller),
read_debug_console,manage_tools,and the daemon's
open/close/resize_session.Both share servers are supported through one
VoiceCallService: the GUI executor invokes the MCPhandlers in-process (so it works even with the user's MCP endpoint disabled, and honors
disabledMcpToolsfor free), while the daemon executor maps ontoDaemonMcpToolswith the smallerheadless surface. The instructions template only describes the tools actually advertised, so the
agent never reaches for
run_commandon a daemon share.Where the settings live
One shared
BossCallingSectionrenders in Settings → Session Sharing and in both sharewindows — the key can be added where the link is being handed out. Its status line mirrors exactly
what the viewer is told (
voiceStatus): off / no API key yet / ready. Without that, amissing key presents as no button at all in the browser, which is undiagnosable from the far end.
Two switches, because the surfaces are different trust boundaries:
voiceCallEnabled— the feature. Defaults on; storing a key is the deliberate opt-in, and withno key the host advertises itself unavailable and the viewer shows no button.
voiceCallShareEnabled— whether REMOTE share viewers may call. Currently defaults on; see theopen policy question below. Turning it off leaves the in-app call working.
voiceShowStatusIndicator— whether the host window's status strip shows the segment, matching theexisting per-indicator toggles for MCP and Sharing.
Testing
:compose-ui:desktopTestis green (566 tests). New coverage:VoiceCallServiceTest(17) — enable/key gating,not_controller, tool-error paths, mint successand 401 against a loopback stub, a 5xx never forwarding OpenAI's error text to the viewer, the
call-token gate (absent/forged/retired/expired), the kill switch stopping a call already running,
mint rate limiting, the in-flight cap admitting exactly four, and refusing a new call at capacity
rather than evicting a live one.
HostVoiceCallControllerTest(8) — the in-app call driven with a fake socket and fake audio: thesession/audio handshake, base64 mic appends, mute stopping the stream, playback and barge-in, a
two-call round emitting exactly one follow-up turn, call-id dedupe, refusing to open a socket
without a key or with the switch off, teardown releasing both, and ending mid-connect never
opening the mic.
GuiVoiceToolExecutorTest(8) — the scope gate (a named out-of-scope tab refused for every tool,a stale focus falling back without ever returning the foreign tab's data), read-only and prefixed
embedder configs, and the argument-less tools answering with nothing focused.
VoiceAgentStorageTest(5) — round-trip, owner-only POSIX perms, corrupt file → null.VoiceToolCatalogTest(3) — OpenAI tools JSON shape, write/guiOnly flags, required params.DaemonVoiceToolExecutorTest(3) —tab_id→session_idmapping, scope rejection, GUI-onlytools not advertised.
ShareProtocolTest— round-trip +"t"discriminator for all 7 new messages.ShareViewerAssetTest— the CSP namesapi.openai.comexactly, no wildcard.ShareViewerScriptTest(the existing Node harness that runs viewer.js) covers the new viewercode loading and initializing.
Manually verified against a running app: the served viewer carries the new CSP and markup, the
settings round-trip persists, and the host correctly advertises
disabled/no_key/ready.Known open policy question: one key and one
voiceCallEnabledtoggle currently gate bothsurfaces, and both default on — so storing a key to try the in-app call also enables voice-driven
run_commandfor anyone already holding a control-share link. The host notification on call start isan after-the-fact signal, not consent. A separate toggle for the remote surface (defaulting off), or
a one-time host confirmation on the first remote call, is worth deciding before merge.
Indirect prompt injection is inherent here: the agent reads scrollback — which an attacker can
influence via
curloutput, a build log, a README — while holdingrun_command. The mitigationtoday is instruction-level only. A host-side approval for the first write tool of a call would be a
proportionate guard.
Not yet verified: a real end-to-end call. The key available for testing returns HTTP 500 from
every OpenAI endpoint — including
GET /v1/modelsand a plaingpt-4o-minicompletion — while adeliberately bogus key gets a clean 401 from the same endpoint, so it is an account-side fault at
OpenAI rather than anything in this branch (the minimal possible mint body fails identically). The
audio path, live tool calls, and the mid-call failure toasts still need a working key before merge.