Skip to content

feat(mistralai): server-side turn detection for Voxtral 3 realtime STT - #8060

Draft
vaflag wants to merge 1 commit into
livekit:mainfrom
vaflag:vaflag/mistralai-voxtral3-turn-detection
Draft

vaflag wants to merge 1 commit into
livekit:mainfrom
vaflag:vaflag/mistralai-voxtral3-turn-detection

Conversation

@vaflag

@vaflag vaflag commented Oct 22, 2026

Copy link
Copy Markdown

Voxtral 3 realtime STT can detect turns on the server. It streams turn.start, turn.eager_end, turn.resume and turn.end events. Today the plugin never sends turn_detection and drops those events, so realtime STT always needs the client-side Silero VAD.

This adds an opt-in turn_detection option that uses the same field names as the Mistral API. When it is set, the plugin sends it in a session.update right after connecting, skips Silero, and maps the events the way the Deepgram Flux plugin does:

  • turn.start to START_OF_SPEECH
  • turn.eager_end to PREFLIGHT_TRANSCRIPT
  • turn.resume to INTERIM_TRANSCRIPT, which lets the pipeline drop a stale preflight
  • turn.end to a flush, whose transcription.done becomes FINAL_TRANSCRIPT and then END_OF_SPEECH
from livekit.agents import AgentSession
from livekit.plugins import mistralai

session = AgentSession(
    stt=mistralai.STT(
        model="voxtral-mini-transcribe-realtime-3",
        turn_detection={"min_silence_ms": 300},
    ),
    turn_handling={"turn_detection": "stt"},
)

The session.update goes through the SDK connection's websocket, because mistralai<3 cannot send turn_detection yet. An explicit vad still wins. The 2602 models reject turn_detection, so they ignore it with a warning. Contradictory thresholds raise early, since the server closes the session on them.

Testing: tests/test_plugin_mistralai_stt.py runs a real SpeechStream against a fake connection (event order, a turn that starts before the previous transcript lands, validation, ignored cases). tests/test_stt.py gets a streaming entry with turn detection, which passed against the public API on voxtral-mini-transcribe-realtime-3 (WER 0.034). I added Mistral to the language exemption in that test because the realtime models return no detected language.

Not covered yet: a full AgentSession run end to end, and speech start/end times on the events. This overlaps with #7796, which touches the same _recv_task and also creates tests/test_plugin_mistralai_stt.py. Happy to rebase on whichever lands first.

Voxtral 3 realtime models can detect turns on the server and stream
turn.start, turn.eager_end, turn.resume and turn.end events. The plugin
did not send turn_detection and dropped those events, so realtime STT
always needed a client VAD.

turn_detection is opt-in, like in the Mistral API. When set, the plugin
sends it in a session.update right after connecting, skips the Silero
VAD, and maps the events the way the Deepgram Flux plugin does:
turn.start to START_OF_SPEECH, turn.eager_end to PREFLIGHT_TRANSCRIPT,
turn.resume to INTERIM_TRANSCRIPT, and turn.end to a flush whose
transcription.done becomes FINAL_TRANSCRIPT plus END_OF_SPEECH.

The session.update goes through the SDK connection's websocket because
mistralai<3 cannot send turn_detection. An explicit vad still wins,
and the 2602 models, which reject turn_detection, ignore it with a
warning. Contradictory thresholds raise early, since the server closes
the session on them.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant