Skip to content

fix(llm): reposition mid-conversation instructions for Gemini gateway and Mistral - #6591

Open
L65FREAD wants to merge 1 commit into
livekit:mainfrom
L65FREAD:fix/openai-fmt-mid-conversation-instructions
Open

fix(llm): reposition mid-conversation instructions for Gemini gateway and Mistral#6591
L65FREAD wants to merge 1 commit into
livekit:mainfrom
L65FREAD:fix/openai-fmt-mid-conversation-instructions

Conversation

@L65FREAD

Copy link
Copy Markdown

Summary

generate_reply(instructions=...) appends the per-turn instructions as a trailing system message. Two serialization paths mishandled it:

  • LiveKit Inference / Gemini: the gateway serializes with the openai format, which passes system messages through as-is — and Gemini can't take a system role after the start of the conversation.
  • Mistral: the Conversations serializer maps every system message onto the instructions request field, so the trailing per-turn message silently overwrote the agent's base prompt.

Both now route through the SDK's existing convert_mid_conversation_instructions (already used by the google/anthropic/aws serializers), which keeps the first system message as the base preamble and rewrites later ones as position-preserving <instructions>-wrapped user turns.

Where the conversion happens:

  • The inference gateway applies it in LLMStream._run at serialization time — so stream.chat_ctx and the llm_request telemetry span keep the original roles, matching the plugins — gated by a model-prefix allowlist (currently google/). OpenAI-compatible gateway providers keep the system role untouched.
  • AIPlatformLLM (Vertex AI Model Garden), which reuses the shared stream with the openai wire format, opts in unconditionally since self-deployed model templates commonly reject non-leading system roles.
  • The mistralai serializer applies it directly, same as google/anthropic/aws.

Helper hardening while touching it:

  • A system message only counts as the preamble when no conversation content precedes it — previously, with no base system message anywhere, a trailing per-turn message stayed a trailing system message.
  • Text-less mid-conversation system messages are dropped instead of leaking through (they could shadow the base prompt on Mistral).

Testing

  • New unit tests: gateway repositioning end-to-end (google model converts, openai model passes through, original ctx not mutated), mistral serializer conversion, the no-preamble edge, and the empty-system-message edge.
  • Full unit suite: 1177 passed (the 9 errors are pre-existing on the base commit too); ruff and full-repo mypy clean.

@L65FREAD
L65FREAD requested a review from a team as a code owner July 28, 2026 20:32
@CLAassistant

CLAassistant commented Jul 28, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants