Repository navigation
Conversation
Voxtral 3 realtime models can detect turns on the server and stream turn.start, turn.eager_end, turn.resume and turn.end events. The plugin did not send turn_detection and dropped those events, so realtime STT always needed a client VAD. turn_detection is opt-in, like in the Mistral API. When set, the plugin sends it in a session.update right after connecting, skips the Silero VAD, and maps the events the way the Deepgram Flux plugin does: turn.start to START_OF_SPEECH, turn.eager_end to PREFLIGHT_TRANSCRIPT, turn.resume to INTERIM_TRANSCRIPT, and turn.end to a flush whose transcription.done becomes FINAL_TRANSCRIPT plus END_OF_SPEECH. The session.update goes through the SDK connection's websocket because mistralai<3 cannot send turn_detection. An explicit vad still wins, and the 2602 models, which reject turn_detection, ignore it with a warning. Contradictory thresholds raise early, since the server closes the session on them.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Voxtral 3 realtime STT can detect turns on the server. It streams
turn.start,turn.eager_end,turn.resumeandturn.endevents. Today the plugin never sendsturn_detectionand drops those events, so realtime STT always needs the client-side Silero VAD.This adds an opt-in
turn_detectionoption that uses the same field names as the Mistral API. When it is set, the plugin sends it in asession.updateright after connecting, skips Silero, and maps the events the way the Deepgram Flux plugin does:turn.starttoSTART_OF_SPEECHturn.eager_endtoPREFLIGHT_TRANSCRIPTturn.resumetoINTERIM_TRANSCRIPT, which lets the pipeline drop a stale preflightturn.endto a flush, whosetranscription.donebecomesFINAL_TRANSCRIPTand thenEND_OF_SPEECHThe
session.updategoes through the SDK connection's websocket, becausemistralai<3cannot sendturn_detectionyet. An explicitvadstill wins. The 2602 models rejectturn_detection, so they ignore it with a warning. Contradictory thresholds raise early, since the server closes the session on them.Testing:
tests/test_plugin_mistralai_stt.pyruns a realSpeechStreamagainst a fake connection (event order, a turn that starts before the previous transcript lands, validation, ignored cases).tests/test_stt.pygets a streaming entry with turn detection, which passed against the public API onvoxtral-mini-transcribe-realtime-3(WER 0.034). I added Mistral to the language exemption in that test because the realtime models return no detected language.Not covered yet: a full
AgentSessionrun end to end, and speech start/end times on the events. This overlaps with #7796, which touches the same_recv_taskand also createstests/test_plugin_mistralai_stt.py. Happy to rebase on whichever lands first.