fix(inference): finalize AvatarSession Rosetta port

[toubatbrian]

Summary

• follows up merged feat(inference): add avatar session #2101, ported from feat(inference): add inference.AvatarSession (avatar provisioning via the gateway) agents#6492
• allows inference.AvatarSession.start() retry when provisioning fails before a create response is parsed
• rolls back only base lifecycle bindings after an ambiguous create failure, preserving any avatar participant the gateway may already have provisioned
• preserves one create idempotency key across internal HTTP retries and later start() retries
• refreshes inference gateway authentication on every create and terminate retry attempt
• falls back to the connected room's server SID when the job dispatch room SID is absent or empty
• deduplicates shutdown callbacks per job context and prevents stale callbacks from an earlier context closing a later successful retry
• preserves successful-create one-shot behavior, nullable IDs, normal aclose() participant removal, and retryable termination state
• preserves a preinstalled avatar audio output through RoomIO startup and wraps it for synchronized transcription without taking ownership
• adds an @livekit/agents patch changeset for the follow-up fixes

Parity revalidation

Compared the actual patches for #2101 and livekit/agents#6492 against merged JS main and current Python. Revalidated request shaping, per-attempt authentication, retries/timeouts, nullable room SID fallback, waitForJoin() ordering, lifecycle and partial-start failure, cleanup/termination retry, double-start, audio rebinding, nullable IDs, examples, exports, and tests.

The follow-up fixes JS-specific integration gaps around failed-start rollback, cross-start idempotency, shutdown callback ownership, ambiguous-create participant preservation, RoomIO handling of an avatar-installed audio output, connected-room SID resolution, and authentication refresh across long retry delays.

The JS whole-request timeout and retrying/escalating termination behavior remain deliberate, tested adaptations from Python's separate total/connect timeouts and best-effort single termination attempt.

Tests

• focused AvatarSession, inference avatar, and RoomIO suites — 3 files, 57 tests passed
• full agents suite — 109 files, 1,465 passed and 5 skipped
• strict TDD regressions covered ambiguous create rollback, stable cross-start idempotency, external RoomIO audio ownership, stale cross-context shutdown callbacks, connected-room SID fallback, and fresh auth with stable idempotency across retries
• pnpm build — 40/40 tasks passed on exact head
• Prettier and changed-file ESLint — passed
• IDE diagnostics and git diff --check — clean
• independent final gates — Spec compliance: APPROVED and Code quality: APPROVED, with no Critical, Important, or Minor findings

Cue voice E2E

• Exact head: 13645af1c1299266de334880f355b3435a523acd
• Fresh session: sid_4411b9daf480 (--mode voice, ended cleanly)
• Predicate: {debug_message(.payload.phase="request_parity_verified",.payload.head="13645af1c1299266de334880f355b3435a523acd",.payload.createVerified=true,.payload.authRefreshed=true,.payload.idempotencyStable=true,.payload.roomIoVerified=true) -> function_tools_executed(.function_calls[0].name="close_avatar") -> debug_message(.payload.phase="terminate_completed",.payload.terminateVerified=true)} and agent_state_changed(.new_state="AS_SPEAKING") and conversation_item_added(.item.message.role="ASSISTANT")
• Result: resolved. Persisted events prove two create attempts used distinct bearer-token hashes and one stable idempotency-key hash; the request carried a non-empty room SID; DataStreamAudioOutput -> SyncedAudioOutput forwarded one frame; termination was verified; and the agent spoke an assistant completion.
• Controlled request evidence: ~/.cue-cli/sessions/sid_4411b9daf480/controlled-gateway-redacted.jsonl contains two create requests around a controlled one-second 503 retry and one terminate request. It persists only URL/method, provider header, Bearer JWT shape, non-secret room/provider session IDs, and SHA-256 token/idempotency representations. No raw bearer, worker, or terminate secret is persisted.
• Request result: create Authorization hashes differ (09c896... then 4fcff5...) while the idempotency-key hash remains 296eff...; terminate used its own freshly minted Authorization hash with the expected provider session and hashed token evidence.
• Voice/provider evidence: Cue voice input, LiveKit Deepgram nova-3 STT, LiveKit turn detector, LiveKit OpenAI gpt-4.1-mini LLM, and LiveKit Cartesia sonic-3 TTS.
• WAVs: session recording.wav is 4,063,596 bytes, stereo 48 kHz, 21.164 s; command recording.wav is 1,785,932 bytes, stereo 48 kHz, 9.302 s. Both channels in both files contain substantial non-zero samples.
• Artifacts inspected: command result, both WAVs, sanitized gateway capture, and worker logs.
• Limitation: this is a protocol-compatible controlled gateway boundary, not a production LemonSlice/gateway E2E, because production avatar access is not enabled for this project. Connected-room fallback is covered deterministically by the focused regression because a normal dispatched Cue job supplies its SID.

Made with Cursor

[changeset-bot]

🦋 Changeset detected

Latest commit: 13645af

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages

Name	Type
@livekit/agents	Patch
@livekit/agents-plugin-anam	Patch
@livekit/agents-plugin-anthropic	Patch
@livekit/agents-plugin-assemblyai	Patch
@livekit/agents-plugin-azure	Patch
@livekit/agents-plugin-baseten	Patch
@livekit/agents-plugin-bey	Patch
@livekit/agents-plugin-cartesia	Patch
@livekit/agents-plugin-cerebras	Patch
@livekit/agents-plugin-deepgram	Patch
@livekit/agents-plugin-did	Patch
@livekit/agents-plugin-elevenlabs	Patch
@livekit/agents-plugin-fishaudio	Patch
@livekit/agents-plugin-google	Patch
@livekit/agents-plugin-hedra	Patch
@livekit/agents-plugin-hume	Patch
@livekit/agents-plugin-inworld	Patch
@livekit/agents-plugin-krisp	Patch
@livekit/agents-plugin-lemonslice	Patch
@livekit/agents-plugin-liveavatar	Patch
@livekit/agents-plugin-livekit	Patch
@livekit/agents-plugin-minimax	Patch
@livekit/agents-plugin-mistral	Patch
@livekit/agents-plugin-mistralai	Patch
@livekit/agents-plugin-neuphonic	Patch
@livekit/agents-plugin-openai	Patch
@livekit/agents-plugin-perplexity	Patch
@livekit/agents-plugin-phonic	Patch
@livekit/agents-plugin-protoface	Patch
@livekit/agents-plugin-resemble	Patch
@livekit/agents-plugin-rime	Patch
@livekit/agents-plugin-runway	Patch
@livekit/agents-plugin-sarvam	Patch
@livekit/agents-plugin-silero	Patch
@livekit/agents-plugin-soniox	Patch
@livekit/agents-plugin-tavus	Patch
@livekit/agents-plugins-test	Patch
@livekit/agents-plugin-trugen	Patch
@livekit/agents-plugin-xai	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[tinalenguyen]

i noticed there was a lip sync issue when audioEnabled is true in outputOptions, here is what claude said for the fix (pre existing this PR but maybe we can bundle it):

Where: room_io.ts start(), output-setup block.

What: two small changes so RoomIO wraps a pre-installed audio output (the avatar's DataStreamAudioOutput) instead of clobbering it:

const externalAudioOutput = this.agentSession.output.audio ?? undefined;

// don't publish a competing track when the avatar already owns audio
if (this.outputOptions.audioEnabled && !externalAudioOutput) {
  this.participantAudioOutput = new ParticipantAudioOutput(...);
}

// sync against our own output, else the avatar's
const audioOutput = this.participantAudioOutput ?? externalAudioOutput;

Why it works: the synchronizer now wraps the avatar output, so frames flow SyncedAudioOutput → DataStreamAudioOutput → avatar while driving the transcript clock. Nothing else changes — the audioOutput getter and line 578 already assign the wrapped output to session.output.audio.

Result: leave audio enabled (drop audioEnabled: false) → avatar works and transcript is synced.

Also need: confirm close() doesn't double-close the avatar output, reword the stale "ignoring" warning, add a room_io.test.ts case, changeset.

[toubatbrian]

Implemented the RoomIO/avatar integration feedback on exact head 1790765b9e19e4e8f43d0570240e7dad0628325a:

• preserves a preinstalled DataStreamAudioOutput instead of publishing a competing participant track
• wraps that output with SyncedAudioOutput for transcript timing
• leaves externally owned audio open during RoomIO shutdown
• rewords the stale warning and adds focused ownership/frame-forwarding regressions
• adds an @livekit/agents patch changeset

Also fixed stale shutdown callbacks from an earlier job context closing a later successful retry.

Verification: 56 focused tests, 1,464 full agents tests, 40/40 build tasks, green exact-head CI, and Cue session sid_7029772d6319 proving DataStreamAudioOutput -> SyncedAudioOutput, one forwarded frame, stable create idempotency, verified termination, assistant speech, and both stereo WAVs. The PR description contains the full sanitized evidence and controlled-gateway limitation.

[toubatbrian]

Closed the two remaining Python-parity gaps on exact head 13645af1c1299266de334880f355b3435a523acd:

• a job with an absent/empty dispatch room SID now falls back to the connected room's server SID
• create and terminate retries now mint fresh gateway authentication per attempt while preserving one create idempotency key

Verification: strict red-green regressions; 57 focused tests; full agents suite 1,465 passed/5 skipped; build 40/40; exact-head CI green; independent Spec compliance: APPROVED and Code quality: APPROVED; zero unresolved threads.

Fresh Cue session sid_4411b9daf480 resolved and persisted redacted evidence proving distinct create Authorization hashes across a controlled one-second retry, one stable idempotency hash, non-empty room SID, DataStreamAudioOutput -> SyncedAudioOutput, one forwarded frame, verified termination, assistant speech, and non-zero stereo audio in both WAVs. The PR description contains the complete evidence and production-avatar limitation.