Skip to content

fix(avatar): restore the previous audio route on AvatarSession.aclose() - #7282

Open
dorukdumlu wants to merge 2 commits into
livekit:mainfrom
dorukdumlu:fix/avatar-aclose-restore-output
Open

dorukdumlu wants to merge 2 commits into
livekit:mainfrom
dorukdumlu:fix/avatar-aclose-restore-output

Conversation

@dorukdumlu

Copy link
Copy Markdown

Summary

Avatar plugins route the agent session's audio to the avatar with output.replace_audio_tail(...) in start(), and aclose() never put it back. When the avatar fails to start (wait_for_join timing out, the app giving up), the session stays pointed at a dead datastream sink and the agent goes silent instead of degrading to regular audio. With the canonical ordering (avatar started before session.start()), the stale route also makes session.start() set room_options.audio_output = False, so no RoomIO audio ever exists to fall back to.

Fixes #7276

What changed

  • AvatarSession gains _attach_audio_output(sink): it installs the route and remembers what replace_audio_tail() swapped out. aclose() restores it:
    • the previous tail goes back under any wrapper chain (TranscriptSynchronizer and friends stay attached),
    • a route that was installed over nothing is cleared, so a later session.start() sets up room audio normally,
    • a route someone else installed after the avatar is left alone (identity-guarded).
  • AgentOutput.replace_audio_tail() now returns the sink it replaced, and AgentOutput.audio_tail exposes the current tail. Both backward compatible.
  • All in-repo avatar plugins are migrated to the helper, a mechanical one-line change each. Out-of-tree plugins that call replace_audio_tail directly keep working exactly as before, just without the restore.

One known limit, called out rather than papered over: if the avatar dies after session.start() already ran, restoring the route can't bring audio back, because RoomIO audio was disabled at startup and there is no _ParticipantAudioOutput to return to. Handling that would mean building the room audio output late, which felt like a separate feature. Happy to take it as a follow-up if there's interest.

Test plan

  • New tests/test_avatar_session_close.py (5 tests): restore over nothing, restore of a previous output, restore of the tail under a wrapper proxy, a newer route wins, aclose without an attach is a no-op. The restore tests fail without the fix.
  • pytest tests/test_audio_sink_proxy.py tests/test_inference_avatar.py tests/test_recorder_io.py --unit (90 passed)
  • ruff and mypy clean on the touched files

Avatar plugins route the agent session's audio to the avatar via
output.replace_audio_tail() in start(), and aclose() never put it back. On a
failed avatar start (wait_for_join timing out, the app giving up on the
avatar), the session was left pointed at a dead datastream sink, so the agent
went silent instead of degrading to regular audio. With the avatar started
before session.start(), the stale route also made session.start() disable
RoomIO audio entirely.

AvatarSession now installs the route through _attach_audio_output(), which
remembers what replace_audio_tail() swapped out, and aclose() restores it:
the previous tail goes back under any wrapper chain, a route installed over
nothing is cleared so a later session.start() sets up room audio normally,
and a route someone else installed after the avatar is left alone. All
in-repo avatar plugins are migrated to the helper (a mechanical one-line
change each); out-of-tree plugins keep working as before, just without the
restore. replace_audio_tail() now returns the sink it replaced, and
AgentOutput.audio_tail exposes the current tail.

Fixes livekit#7276
devin-ai-integration[bot]

This comment was marked as resolved.

…-safe

Two review findings. The in-repo sweep missed the one avatar that lives in
core rather than livekit-plugins: the inference AvatarSession still installed
its route through replace_audio_tail directly, so its aclose() had nothing
recorded to restore and the dead datastream sink survived. It now goes
through _attach_audio_output like every plugin.

And the restore sat behind the participant-removal await, so a cancellation
mid-close (a job-shutdown deadline) skipped it. The restore is synchronous
and independent of the removal, so it now runs first, before any await.
@dorukdumlu

Copy link
Copy Markdown
Author

Both findings were right, addressed in acc2670. The red one was a plain miss on my side: I swept livekit-plugins for replace_audio_tail callers and forgot the one avatar that lives in core, so the inference AvatarSession kept bypassing the tracking. It now goes through _attach_audio_output like the plugins, with a close regression test on the inference path. For the yellow, the restore is synchronous and doesn't depend on the participant removal at all, so it simply runs first in aclose(), before any await; added a test that cancels the close mid-removal and checks the route still came back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AvatarSession.aclose() doesn't unbind the audio output start() installed, so a failed avatar start silently mutes the agent

1 participant