fix(livekit-agents): forward prewarm to the primary LLM in the fallback adapter - #6582
Open
fxhxdxd wants to merge 2 commits into
Open
fix(livekit-agents): forward prewarm to the primary LLM in the fallback adapter#6582fxhxdxd wants to merge 2 commits into
fxhxdxd wants to merge 2 commits into
Conversation
…ck adapter `LLM.prewarm()` is a no-op unless the class overrides `_prewarm_impl`, so the LLM `FallbackAdapter` — which does not — silently swallowed the prewarm call issued by `AgentSession.__init__` and `AgentActivity`. Providers that implement prewarming (e.g. Google) were therefore never warmed when used behind the adapter, losing the time-to-first-token benefit on the first reply. Delegate to the first LLM instance, mirroring `tts.FallbackAdapter.prewarm()`. Only the primary is prewarmed, since the remaining instances are not expected to serve traffic unless it fails. The `loop` argument is forwarded so the prewarm task is still scheduled on the session's event loop. Fixes livekit#6572
chenghao-mou
approved these changes
Jul 28, 2026
chenghao-mou
left a comment
Member
There was a problem hiding this comment.
lgtm. One small test nit.
Comment on lines
+48
to
+49
| loop = asyncio.get_running_loop() | ||
| fallback_adapter.prewarm(loop=loop) |
Member
There was a problem hiding this comment.
nit: even if we drop loop=loop, it will pass as it defaults to the running loop. We can test it with a new loop instead:
class RecordingLLM(FakeLLM):
def __init__(self) -> None:
super().__init__()
self.prewarm_loop: asyncio.AbstractEventLoop | None = None
def prewarm(self, *, loop: asyncio.AbstractEventLoop | None = None) -> None:
self.prewarm_loop = loop
...
supplied_loop = asyncio.new_event_loop()
try:
fallback_adapter.prewarm(loop=supplied_loop)
assert primary.prewarm_loop is supplied_loop
finally:
supplied_loop.close()
Author
There was a problem hiding this comment.
Good catch, you're right that the original assertion held either way, since the wrapped LLM defaults to the running loop when loop is omitted.
Switched to recording the loop the primary is asked to prewarm on and supplying one distinct from the running loop, per your suggestion. Verified it now fails if loop=loop is dropped from the adapter, which the previous version did not catch. Pushed in aeb0b79.
…nt loop The previous assertion passed even without forwarding `loop`, since the wrapped LLM falls back to the running loop. Record the loop the primary is asked to prewarm on and supply one distinct from the running loop, so dropping the argument fails the test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6572.
Problem
LLM.prewarm()short-circuits unless the concrete class overrides_prewarm_impl(livekit-agents/livekit/agents/llm/llm.py:162-163).llm.FallbackAdapterdoesn't override it, so everyprewarm()call it receives — fromAgentSession.__init__(voice/agent_session.py:490) and fromAgentActivity(voice/agent_activity.py:636,721) — returns immediately without ever reaching the wrapped instances. Providers that do implement prewarming are therefore never warmed when used behind the adapter, so the first reply pays the full DNS + TLS setup cost.tts.FallbackAdapteralready handles this correctly (tts/fallback_adapter.py:126-128); the LLM adapter is the gap.stt.FallbackAdapterhas the same gap — happy to cover it in this PR or a follow-up, whichever you prefer.Fix
Override
prewarm()to delegate to the first instance, mirroring the TTS adapter. Only the primary is prewarmed: the remaining instances are not expected to serve traffic unless it fails, and warming them would open connections to providers that may never be used.The
loopargument is forwarded, which is the one place this differs from the TTS version —LLM.prewarm()acceptsloop(the TTS and STT variants don't) andAgentSessionpasses the session loop explicitly, so dropping it would schedule the prewarm task on the wrong loop.Tests
New
tests/test_llm_fallback.py:test_prewarm_forwarded_to_primary_llm— the primary's_prewarm_implruns and the fallback instance stays cold.test_prewarm_forwards_event_loop— the prewarm task is scheduled on the loop supplied by the caller.Both fail on
mainand pass with this change.ruff check,ruff format --checkandmypy --strictare clean on the touched files, andtest_llm_fallback.py,test_tts_fallback.pyandtest_stt_fallback.pypass together (15 passed).