Skip to content

feat(voice): delegate reasoning and tools to a second LLM - #6602

Draft
longcw wants to merge 1 commit into
mainfrom
longc/agent-delegation
Draft

feat(voice): delegate reasoning and tools to a second LLM#6602
longcw wants to merge 1 commit into
mainfrom
longc/agent-delegation

Conversation

@longcw

@longcw longcw commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Splits an agent into a conversation model that owns the dialogue and a delegation_llm that does the reasoning and tool work and never speaks. Realtime and low-latency models are good at conversation and weak at tool calling; frontier text models are the reverse, and today an agent has to pick one and accept the other half.

session = AgentSession(
    llm=openai.realtime.RealtimeModel(),   # owns turn-taking, gets one tool
    delegation_llm="openai/gpt-5.5",       # gets lookup_order and issue_refund
)
agent = Agent(instructions="...", tools=[lookup_order, issue_refund])

How it works

Delegation is an ordinary tool call β€” one builtin lk_agents_delegate β€” so it reuses the async tool lifecycle rather than adding a parallel one: tracing, duplicate handling, cancellation, and the deferred reply that carries the answer back.

The call releases the turn immediately, and its dispatch note being answered is what acknowledges to the user; the answer arrives later as its own reply. Relying on the model to speak alongside the call does not work β€” realtime models routinely emit the call and no speech.

Each side only sees tool traffic for its own tools. The delegation's calls stay out of the conversation entirely, and the delegate pairs stay out of the delegation's context, so neither model reads a tool it cannot call.

API

  • AgentSession(delegation_llm=..., delegation_options=...), same on Agent, agent wins per key
  • Agent.delegation_node(task, ctx, chat_ctx, tools, model_settings) overrides the loop; returns the answer, or None to answer without speaking
  • ToolFlag.NO_DELEGATE keeps a latency-critical tool on the conversation model
  • RunContext.update(..., silent=True) releases a tool without generating a reply

Behaviour changes outside delegation

  • _deliver_reply no longer generates a reply when the model has auto_tool_reply_generation β€” it continues on its own. This affects async tools on google, aws, phonic and ultravox, and is worth a manual check on those providers.
  • A tool can no longer be both CANCELLABLE and NO_DELEGATE; the combination is contradictory and now raises at decoration time.

An agent can split into a conversation model that talks to the user and a
delegation LLM that does the reasoning and tool work. The conversation model is
offered a single builtin `lk_agents_delegate` tool; every other tool goes to the
delegation LLM unless flagged NO_DELEGATE.

Delegation is expressed as an ordinary tool call, so it reuses the async tool
lifecycle rather than adding a parallel one: tracing, duplicate handling,
cancellation, and the deferred reply that carries the answer back. Delegated
tools run on a session-scoped executor and their progress is re-attributed to
the delegate call, so each side only sees tool traffic for its own tools.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant