Skip to content

fix(livekit-agents): forward prewarm to the primary LLM in the fallback adapter - #6582

Open
fxhxdxd wants to merge 2 commits into
livekit:mainfrom
fxhxdxd:fix/llm-fallback-adapter-prewarm
Open

fix(livekit-agents): forward prewarm to the primary LLM in the fallback adapter#6582
fxhxdxd wants to merge 2 commits into
livekit:mainfrom
fxhxdxd:fix/llm-fallback-adapter-prewarm

Conversation

@fxhxdxd

@fxhxdxd fxhxdxd commented Jul 28, 2026

Copy link
Copy Markdown

Fixes #6572.

Problem

LLM.prewarm() short-circuits unless the concrete class overrides _prewarm_impl (livekit-agents/livekit/agents/llm/llm.py:162-163). llm.FallbackAdapter doesn't override it, so every prewarm() call it receives — from AgentSession.__init__ (voice/agent_session.py:490) and from AgentActivity (voice/agent_activity.py:636,721) — returns immediately without ever reaching the wrapped instances. Providers that do implement prewarming are therefore never warmed when used behind the adapter, so the first reply pays the full DNS + TLS setup cost.

tts.FallbackAdapter already handles this correctly (tts/fallback_adapter.py:126-128); the LLM adapter is the gap.

stt.FallbackAdapter has the same gap — happy to cover it in this PR or a follow-up, whichever you prefer.

Fix

Override prewarm() to delegate to the first instance, mirroring the TTS adapter. Only the primary is prewarmed: the remaining instances are not expected to serve traffic unless it fails, and warming them would open connections to providers that may never be used.

The loop argument is forwarded, which is the one place this differs from the TTS version — LLM.prewarm() accepts loop (the TTS and STT variants don't) and AgentSession passes the session loop explicitly, so dropping it would schedule the prewarm task on the wrong loop.

Tests

New tests/test_llm_fallback.py:

  • test_prewarm_forwarded_to_primary_llm — the primary's _prewarm_impl runs and the fallback instance stays cold.
  • test_prewarm_forwards_event_loop — the prewarm task is scheduled on the loop supplied by the caller.

Both fail on main and pass with this change. ruff check, ruff format --check and mypy --strict are clean on the touched files, and test_llm_fallback.py, test_tts_fallback.py and test_stt_fallback.py pass together (15 passed).

…ck adapter

`LLM.prewarm()` is a no-op unless the class overrides `_prewarm_impl`, so the
LLM `FallbackAdapter` — which does not — silently swallowed the prewarm call
issued by `AgentSession.__init__` and `AgentActivity`. Providers that implement
prewarming (e.g. Google) were therefore never warmed when used behind the
adapter, losing the time-to-first-token benefit on the first reply.

Delegate to the first LLM instance, mirroring `tts.FallbackAdapter.prewarm()`.
Only the primary is prewarmed, since the remaining instances are not expected to
serve traffic unless it fails. The `loop` argument is forwarded so the prewarm
task is still scheduled on the session's event loop.

Fixes livekit#6572
@fxhxdxd
fxhxdxd requested a review from a team as a code owner July 28, 2026 13:24
@CLAassistant

CLAassistant commented Jul 28, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

@chenghao-mou chenghao-mou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. One small test nit.

Comment thread tests/test_llm_fallback.py Outdated
Comment on lines +48 to +49
loop = asyncio.get_running_loop()
fallback_adapter.prewarm(loop=loop)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: even if we drop loop=loop, it will pass as it defaults to the running loop. We can test it with a new loop instead:

class RecordingLLM(FakeLLM):
    def __init__(self) -> None:
        super().__init__()
        self.prewarm_loop: asyncio.AbstractEventLoop | None = None

    def prewarm(self, *, loop: asyncio.AbstractEventLoop | None = None) -> None:
        self.prewarm_loop = loop

...
supplied_loop = asyncio.new_event_loop()
try:
    fallback_adapter.prewarm(loop=supplied_loop)
    assert primary.prewarm_loop is supplied_loop
finally:
    supplied_loop.close()

@fxhxdxd fxhxdxd Jul 28, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, you're right that the original assertion held either way, since the wrapped LLM defaults to the running loop when loop is omitted.

Switched to recording the loop the primary is asked to prewarm on and supplying one distinct from the running loop, per your suggestion. Verified it now fails if loop=loop is dropped from the adapter, which the previous version did not catch. Pushed in aeb0b79.

…nt loop

The previous assertion passed even without forwarding `loop`, since the wrapped
LLM falls back to the running loop. Record the loop the primary is asked to
prewarm on and supply one distinct from the running loop, so dropping the
argument fails the test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LLM FallbackAdapter doesn't pass through prewarm to contained LLM instances

3 participants