Add agent stt sdk - #127
Open
giorgosHadji wants to merge 37 commits into
Open
Conversation
giorgosHadji
commented
Aug 18, 2026
giorgosHadji
commented
Aug 18, 2026
giorgosHadji
commented
Aug 18, 2026
giorgosHadji
commented
Aug 18, 2026
giorgosHadji
commented
Aug 18, 2026
giorgosHadji
commented
Aug 18, 2026
giorgosHadji
marked this pull request as draft
August 18, 2026 11:20
giorgosHadji
commented
Aug 18, 2026
giorgosHadji
commented
Aug 18, 2026
giorgosHadji
commented
Aug 18, 2026
giorgosHadji
marked this pull request as ready for review
August 18, 2026 13:01
giorgosHadji
commented
Aug 18, 2026
speechmatics-agent-stt is an extension of the RT SDK for the Voice Agent Service. The service works in segments rather than word groups and reports speech and turn events, so the client consumes AddSegment/AddPartialSegment and accumulates the session transcript from them. The SDK runs no VAD or turn detection of its own. Either the service's VAD closes turns (VADMode.SERVER), or the host framework's does (VADMode.CLIENT) by calling finalize(), which sends ForceEndOfUtterance stamped with the audio position at the moment of the call. This is what Pipecat and LiveKit need, and replaces the voice SDK's in-process VAD and smart-turn models. - endpoint is the RT URL plus /agent, optionally plus a service profile - TranscriptionConfig adds vad_mode, vad_config and emit_sentences, and leaves model unset so it cannot conflict with the profile's locked operating_point - every server message is kept in client.events, including unmodelled ones - 16 kHz raw PCM defaults, as the service requires Verified against the real service with a stub transcriber behind it. Nothing in the voice SDK or Pipecat is touched; see sdk/agent_stt/PLAN.md for the Pipecat migration plan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Engine silence-based end of utterance is off for the Agent STT service, so TurnDetectionMode.FIXED has nothing to map onto and is removed rather than aliased. Records the settings that go with it, and notes in the SDK README that a turn ends only via the service VAD or finalize(). SMART_TURN stays open: it ran an in-process model, which this SDK does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The service has no smart-turn endpoint yet and this SDK loads no models, so the mode cannot be honoured. It costs nothing: any host-side endpointing, including Pipecat's own turn analyzer, reaches the service through finalize(). Makes VADMode.CLIENT explicit that it is agnostic about what produced the end-of-speech signal - VAD, turn model, or push-to-talk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A typed entry for transcription_config.additional_vocab, so callers can pass words with pronunciation hints instead of raw dicts. Both forms are accepted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The SDK previously left `model` unset, on the reasoning that the service's default profile pins and locks `operating_point: enhanced`, so sending a model name alongside it would put both keys in the merged StartRecognition. That reasoning held for a direct connection to the service websocket, but it is not how requests actually reach it: they go through a proxy first, which resolves the Agent STT model name onto the engine's operating point. The transcriber therefore never sees a name it has no notion of, and sending `linden-1` is safe. Replace the re-exported RT `Model` with an Agent STT one. The RT models (`enhanced`, `standard`) are not Agent STT models, so leaving them reachable from this package only invited configs the service would reject. Default resolution uses the same `_UNSET` sentinel pattern as `rt.TranscriptionConfig` rather than a plain default, because the deprecated `operating_point` has to suppress the model rather than collide with it: passing `operating_point` leaves `model` unsent, and passing both still raises. `DEFAULT_MODEL` is a separate constant and the docstrings name it rather than the value, so linden-2 is one enum member plus, if it becomes the default, one line. The end-to-end verification recorded in PLAN.md predates this and ran against the service with a stub transcriber and no proxy, so it exercised neither the model default nor its resolution. Noted there as still needing a run against the real proxy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
added 8 commits
August 26, 2026 10:50
| Raises: | ||
| ConnectionError: If the WebSocket connection fails. | ||
| TimeoutError: If the service does not accept the session in time. | ||
| """ |
Contributor
There was a problem hiding this comment.
This forwards the passed transcription_config, but send_message builds turn_config from self._config, which is still the constructor's. So:
client = AgentSttAsyncClient(config=TranscriptionConfig(turn_detection_mode=TurnDetectionMode.EXTERNAL))
await client.start_session(transcription_config=TranscriptionConfig(turn_detection_mode=TurnDetectionMode.VAD))
sends transcription_config from the argument and turn_config from the constructor, and the session silently runs in the wrong turn mode.
Looks like transcribe() avoids this by assigning self._config before starting; start_session should do the same, or the injection should read the config that is actually being sent.
Contributor
Author
There was a problem hiding this comment.
Well the issue here is that we have both, if the client init the AgentSttAsyncClient with one and then one start_session uses another its their own fault. This issues could happen with other parameters I am sure. Its bad that we allow this, it should have been only one place.
similarly for AsyncClient (rt one)
Contributor
|
For FEOU, the Agent STT service requires the timestamp (stricter than RT) - so there's a few changes needed to handle it, I think (either dropping the None branch so a timestamp is always sent, or raising client-side so the caller finds out at the call rather than from a Warning several messages later). |
Contributor
|
All comments posted. Once resolved, happy to approve! |
| if self._is_connected: | ||
| return | ||
|
|
||
| await self.start_session( |
Contributor
There was a problem hiding this comment.
Should turn_config appear here too?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add agent stt sdk which will communicate with our AgentSTT offering - add docs here.
This MR doesn't remove/deprecate the voice sdk - that will happen in follow up MRs.
How to test this:
Get an api key, and point to rt preview (SPEECHMATICS_RT_URL=wss://preview.rt.speechmatics.com/v2)
Install locally the agent stt sdk - >
python -m pip install -e sdk/agent_stt[dev]Then run for a file based test :
python examples/agent_stt/realtime_file/main.pyor you can use an interactive script (using microphone):
python .\examples\agent_stt\microphone\main.py