feat(inference): add avatar session

[rosetta-livekit-bot]

What

Ports livekit/agents#6492 to agents-js by adding inference.AvatarSession, a LiveKit Inference gateway-provisioned avatar session for LemonSlice and future avatar providers.

Changes

• Adds agents/src/inference/avatar.ts with gateway create/terminate calls, local avatar worker token minting, retry/idempotency behavior, terminate token handling, response sample-rate audio output rebinding, double-start guarding, and standalone/job-context credential and room resolution.
• Exports AvatarSession, LemonSliceOptions, and related types from inference.
• Ports the source unit coverage to agents/src/inference/avatar.test.ts.
• Adds a JS inference avatar example and documents the Inference variant in the LemonSlice README.
• Adds a minor changeset for @livekit/agents.

No known target infrastructure gap remains. One agents-js adaptation was needed: before a JS room is connected, Room.name can be unset, so AvatarSession.start() falls back to JobContext.job.room.name and JobContext.job.room.sid in job mode; standalone mode uses room.sid when available or looks up the room via RoomServiceClient.

Source diff coverage

Source file	Classification	Target handling
examples/avatar/README.md	Adapted	Ported the Inference variant note to plugins/lemonslice/README.md, because agents-js does not have an examples/avatar/ directory and the LemonSlice plugin README is the existing target-side avatar docs counterpart.
examples/avatar/inference_agent.py	Adapted	Ported to examples/src/inference_avatar.ts using agents-js defineAgent, AgentSession, Agent, and object-style inference model constructors. The behavior, env vars, and Inference avatar flow are preserved.
livekit-agents/livekit/agents/inference/__init__.py	Ported	Ported to agents/src/inference/index.ts by exporting AvatarSession, LemonSliceOptions, parseAvatarModel, and related public types. The Python lazy PEP 562 export is not applicable to the TypeScript barrel because the target inference package does not need that circular-import workaround.
livekit-agents/livekit/agents/inference/avatar.py	Adapted	Ported to agents/src/inference/avatar.ts. Python aiohttp and context APIs were adapted to fetch/AbortSignal, livekit-server-sdk AccessToken, agents-js voice.AvatarSession/DataStreamAudioOutput, camelCase public constructor option names where the target convention already uses them, and JS room/job context shapes.
tests/test_inference_avatar.py	Ported	Ported to agents/src/inference/avatar.test.ts with Vitest and fetch fakes, preserving the source coverage areas: model parsing, credential fallback, payload/header/idempotency behavior, retry/error handling, sample-rate rebinding, worker JWT grants/attributes, double-start guard, terminate behavior, and standalone/job room paths.

Validation

• pnpm vitest run agents/src/inference/avatar.test.ts passes: 29 tests.
• pnpm vitest run agents passes: 104 files, 1393 tests, 5 skipped.
• pnpm vitest run plugins/lemonslice passes: 2 files, 9 tests.
• pnpm build passes.
• pnpm --filter @livekit/agents lint passes with pre-existing warnings only.
• pnpm format:check passes.
• cue-cli text-mode smoke with a temporary dispatchable JS worker passed, asserting an assistant conversation_item_added event after inference.AvatarSession.start() with a mocked gateway response.

Known unrelated validation failures:

• pnpm vitest run examples currently fails in existing examples/src/testing/agent_task.test.ts (No FunctionCallOutputEvent matching criteria found) and reports existing FakeLLM unhandled errors from survey_agent.test.ts. The new examples/src/inference_avatar.ts builds successfully via pnpm build.
• Root pnpm api:check fails before this change's API surface can be checked because @livekit/agents-plugin-liveavatar has no api-extractor.json. Direct pnpm --filter @livekit/agents api:check is also blocked by the repo's existing API Extractor limitation on export * as syntax in agents/dist/index.d.ts.

Source PR: livekit/agents#6492

---

Ported from livekit/agents#6492

Original PR description

What

Adds livekit.agents.inference.AvatarSession — a thin subclass of voice.avatar.AvatarSession that provisions an avatar provider session through the LiveKit Inference gateway (POST /v1/avatar/sessions) instead of a BYOK provider key.

from livekit.agents import AgentSession, inference

avatar = inference.AvatarSession("lemonslice", image_url="https://…", prompt="…")
await avatar.start(session, room=ctx.room)
await avatar.wait_for_join()

Why

BYOK avatar plugins require a customer provider key and bill LiveKit nothing for the avatar. Routing provisioning through Inference gives a single-key experience (LIVEKIT_API_KEY / LIVEKIT_API_SECRET only) and lets LiveKit meter avatar usage — matching inference.STT/LLM/TTS. BYOK plugins stay as-is; this is an additive path.

How

• The agent mints the avatar worker's room token locally (identical to the BYOK plugins — kind=agent, room_join, lk.publish_on_behalf, plus an lk.avatar_provider attribute for future metering) and sends a normalized payload to the gateway; the gateway calls the provider with LiveKit's wholesale key. Media/lip-sync still flow in-room over DataStreamAudioOutput — unchanged from BYOK.
• Model string "lemonslice" or "lemonslice/<agent_id>"; image_url / prompt / idle_prompt first-class; extra_kwargs for provider-specific fields.
• Idempotency-Key per start() (stable across retries) so a retried create replays on the gateway instead of creating a second paid session.
• aclose() terminates the provider session via the gateway (/v1/avatar/sessions/terminate) so it stops billing on shutdown rather than lingering to the idle timeout.
• Uses the gateway-reported sample_rate for DataStreamAudioOutput (no per-provider hardcoding). Works inside an agent job or standalone (derives identity/room-sid from the connected room when there's no job context).
• Lazy PEP 562 export in inference/__init__.py to avoid a circular import during voice-package init.

Files

• livekit-agents/livekit/agents/inference/avatar.py (new)
• livekit-agents/livekit/agents/inference/__init__.py — lazy AvatarSession export
• examples/avatar/inference_agent.py (new) + a note in examples/avatar/README.md
• tests/test_inference_avatar.py (new)

Testing

pytest tests/test_inference_avatar.py — 19 passing: model-string parsing, credential resolution/env fallbacks, create payload/headers (incl. stable Idempotency-Key), retry/error mapping (403 non-retryable, 502 retried), response sample_rate → DataStreamAudioOutput, and aclose() terminate (+ no-op when no session). ruff clean.

Dependencies / rollout

• Requires the gateway endpoints from livekit/agent-gateway#1034 and the avatar_lemonslice feature flag enabled for the project (staging first). Until then, start() returns a 403 not_enabled.
• E2E coverage: livekit/e2e#1255 (agents/avatar/ suite), which activates once this ships and the flag is on.

Refs: LKINF-386

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: 9c0e24c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages

Name	Type
@livekit/agents	Patch
@livekit/agents-plugin-anam	Patch
@livekit/agents-plugin-anthropic	Patch
@livekit/agents-plugin-assemblyai	Patch
@livekit/agents-plugin-azure	Patch
@livekit/agents-plugin-baseten	Patch
@livekit/agents-plugin-bey	Patch
@livekit/agents-plugin-cartesia	Patch
@livekit/agents-plugin-cerebras	Patch
@livekit/agents-plugin-deepgram	Patch
@livekit/agents-plugin-did	Patch
@livekit/agents-plugin-elevenlabs	Patch
@livekit/agents-plugin-fishaudio	Patch
@livekit/agents-plugin-google	Patch
@livekit/agents-plugin-hedra	Patch
@livekit/agents-plugin-hume	Patch
@livekit/agents-plugin-inworld	Patch
@livekit/agents-plugin-krisp	Patch
@livekit/agents-plugin-lemonslice	Patch
@livekit/agents-plugin-liveavatar	Patch
@livekit/agents-plugin-livekit	Patch
@livekit/agents-plugin-minimax	Patch
@livekit/agents-plugin-mistral	Patch
@livekit/agents-plugin-mistralai	Patch
@livekit/agents-plugin-neuphonic	Patch
@livekit/agents-plugin-openai	Patch
@livekit/agents-plugin-perplexity	Patch
@livekit/agents-plugin-phonic	Patch
@livekit/agents-plugin-protoface	Patch
@livekit/agents-plugin-resemble	Patch
@livekit/agents-plugin-rime	Patch
@livekit/agents-plugin-runway	Patch
@livekit/agents-plugin-sarvam	Patch
@livekit/agents-plugin-silero	Patch
@livekit/agents-plugin-soniox	Patch
@livekit/agents-plugin-tavus	Patch
@livekit/agents-plugins-test	Patch
@livekit/agents-plugin-trugen	Patch
@livekit/agents-plugin-xai	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[shawnfeldman]

PR Review Summary

Reviewed the diff, cross-checked the wire contract against agent-gateway's actual pkg/handler/avatar.go (this repo), and verified the two most consequential claims independently against upstream source (livekit/node-sdks) rather than trusting review-agent output at face value.

Good news first: this port does not repeat the Critical bug that hit the source Python PR (livekit/agents#6492) before it merged — terminate_token is correctly captured from the create response (avatar.ts:250) and aclose() only calls terminate when both provider_session_id and terminate_token are present (avatar.ts:293), matching the gateway's all-three-required validation in handleTerminate. Field-by-field, the create/terminate payloads match the gateway's avatarRequest/avatarTerminateRequest structs exactly.

A few other things turned up, though:

Important Issues (should fix)

1. resolveRoomSid checks a Room.sid property that doesn't exist on @livekit/rtc-node, so it always takes the slow RPC path — avatar.ts:474-486

   const maybeRoom = room as Room & { sid?: string | Promise<string> };
   if (typeof maybeRoom.sid === 'string') return maybeRoom.sid;
   if (maybeRoom.sid) return await maybeRoom.sid;
   const client = new RoomServiceClient(livekitUrl, livekitApiKey, livekitApiSecret);
   const rooms = await client.listRooms([roomName]);
   return rooms[0]?.sid;

   Checked livekit/node-sdks (packages/livekit-rtc/src/room.ts) directly: Room has no public sid property, only private sidPromise + async getSid(): Promise<string>. The as Room & { sid?: ... } cast just satisfies the type-checker — at runtime maybeRoom.sid is always undefined, so both fast-path branches are dead and every standalone (non-job-context) start() unconditionally makes an extra RoomServiceClient.listRooms() call that room.getSid() would avoid. Worse, room_sid is optional server-side (json:"room_sid,omitempty"), but this client turns a failure of that avoidable RPC into a hard failure of the whole avatar session (if (!roomSid) throw at line 231). The test doubles mask this — FakeConnectedRoom defines sid as a plain string field, which the real SDK class doesn't have.
   Fix: use await room.getSid() as the primary source (falling back to RoomServiceClient.listRooms only if it returns ''), and update the test doubles to implement getSid() instead of a sid field.

2. connOptions.timeoutMs is a documented, public option that's silently ignored — avatar.ts:31, avatar.ts:425
   AvatarSessionOptions.connOptions: APIConnectOptions documents timeoutMs as the per-request connect timeout, but avatar.ts only ever reads maxRetry/retryIntervalMs from it — the actual abort timeout is the hardcoded module constant REQUEST_TIMEOUT_MS = 60000, used unconditionally in postJson's AbortSignal.timeout(REQUEST_TIMEOUT_MS). Sibling code in this same package (inference/llm.ts) correctly threads this.connOptions.timeoutMs through. A caller who passes a short timeoutMs expecting fail-fast behavior silently gets 60s instead.
   Fix: signal: AbortSignal.timeout(this.connOptions.timeoutMs), drop the now-unused constant.

3. aclose()'s terminate call has no retry (unlike create) and unconditionally swallows every failure, bypassing the framework's own shutdown-failure escalation — avatar.ts:289-311, avatar.ts:394-405
   _createSession retries up to connOptions.maxRetry times with backoff; _terminateSession makes exactly one attempt. aclose()'s catch logs a warn and returns normally for any error — a DNS blip, a 502/503/504, a JWT-signing failure — with no discrimination. That's a real cost here specifically because retrying terminate is safe (the gateway's own hop to the provider already retries teardown and treats 404/410 as success — confirmed in pkg/provider/avatar/lemonslice/lemonslice.go in this repo), and because aclose() never rejecting means the job-shutdown callback's existing logger.error('error while shutting down the job') safety net (in ipc/job_proc_lazy_main.ts) never engages for a terminate failure. In the shipped example, aclose() is never called directly — it only runs once, automatically, at job shutdown — so the warning's own suggested remedy ("call aclose() again") has no real trigger in the documented usage pattern. Net effect: one transient failure at shutdown silently leaks one idle-timeout window of billing with no escalation beyond a warn log on an already-exiting process.
   Fix: give _terminateSession the same retry/backoff _createSession has, and after exhausting retries either let the rejection propagate (so the existing shutdown error-logging fires) or log at error with real escalation, not just warn-and-continue.

4. TOCTOU race in the double-start guard — avatar.ts:190 vs. avatar.ts:250
   The guard (if (this._sessionId !== null || this._providerSessionId !== null) throw) is checked synchronously at the top of start(), but the fields it checks aren't set until after several await points (worker-token mint, room-sid resolution, the gateway create call itself). Two overlapping, unawaited calls to start() on the same instance both pass the guard and both create a real, billed provider session — whichever response is written last wins the instance's fields, and the other session's ids are never stored anywhere, making it permanently un-terminable via this class. The existing "start twice" test only covers the sequential/awaited case, so this has no coverage.
   Fix: set a synchronous _started flag (or small state enum) as the very first statement in start(), before any await, rather than inferring "already started" from response data that arrives after the fact.

5. Gateway's Retry-After hint is never read by the retry loop — avatar.ts:355-390
   The gateway sets Retry-After on 429 (60s) and provider-503-capacity responses, and 429 is one of the three statuses _exceptions.ts marks retryable — but postJson never reads response.headers.get('Retry-After'), so the retry loop always waits intervalForRetry (~2s default) regardless. With default maxRetry: 3, the client exhausts its whole retry budget in ~6s against a server that explicitly asked for 60s, and fails a request that likely would have succeeded.
   Fix: capture Retry-After in the thrown error and prefer it over intervalForRetry when present.

Suggestions

• _createSession/_terminateSession have no access modifier (should be private, matching the treatment already given to _sessionId/_providerSessionId/_terminateToken in the same class) — @internal is a real, working typedoc-exclusion convention here, not a runtime boundary, so this is a "make it actually private too" nit rather than a misrepresentation.
• A create response of {} (no session_id/provider_session_id) would also defeat the double-start guard, since both are optional in CreateSessionResponse. In practice the real gateway's session_id field is not omitempty (always present on a 201), so this can't happen against the actual server today — but the client shouldn't rely on that guarantee holding forever.
• ATTRIBUTE_PUBLISH_ON_BEHALF in types.ts duplicates an existing constant of the same name/value already exported from constants.ts and consumed by voice/room_io/room_io.ts — two sources of truth for the same wire string; avatar.ts should import the existing one instead of redeclaring it.
• A JSON-parse failure on a "successful" (200) response gets folded into the same generic retryable APIConnectionError as a real network error (toAPIError's fallback), so a deterministic server bug and a flaky connection look identical in logs and both burn the retry budget.
• The "no terminate_token, will bill until idle timeout" fact is logged at warn in start() (line ~262) but only debug for the equivalent case in aclose() (line ~304) — worth matching severity so it's visible at both ends of the session's life.
• Test coverage: start() and aclose() are never exercised as one pipeline (every aclose() test hand-sets _providerSessionId/_terminateToken directly rather than letting them flow from a real start() call); the documented "call aclose() again to retry a failed terminate" recovery path is never simulated; resolveRoomSid's RoomServiceClient fallback branch is untested (every fixture room has a plain-string .sid, which — per finding #1 — doesn't reflect the real SDK shape); none of start()'s three defensive throws are triggered by any test; no test asserts on logger content, so a regression that silently drops one of the billing-risk warnings wouldn't be caught.

Strengths

• The terminate_token capture-and-gate logic is correct and doesn't repeat the sibling Python PR's pre-fix Critical bug — verified field-by-field against the gateway's actual Go structs.
• Idempotency-Key generation is correct: one key per _createSession() call, stable across all retry attempts, with a dedicated test.
• _providerSessionId/_terminateToken are captured before the later DataStreamAudioOutput assignment, so a downstream throw in start() doesn't orphan a session that was actually created — a real, tested exception-safety property.
• aclose() is safe to call from every reachable state (never started, started-without-a-token, terminate-already-failed, already-terminated).
• Construction-time validation (missing credentials, avatar_id/image_url mutual exclusivity) fails fast with clear errors before any network call is possible.