Repository navigation
fix: report speech end timestamps across stt plugins - #7796
chenghao-mou wants to merge 29 commits into
Conversation
The default stt_node offsets provider timestamps from the session start, but the recognizer converted them with the pipeline's first audio frame. On a pipeline created after a handoff the two bases differ by the session's age, so stopped_speaking_at landed in the future and was clamped to the arrival time. The stream now absorbs the delay between its creation and its first frame, and re-anchors on the next frame after a reconnect, so its offset is exact on the caller's timeline. The default node records the base it measured from on the pipeline, and the recognizer adds that same base. A custom node that forwards provider timestamps untouched is anchored on the pipeline's creation, when it opens its stream. Addresses AGT-3678 Co-authored-by: AndraΕΎ Pajtler <1963904+AndrazP@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
β¦irst-audio based Two review findings. A frame queued while a connection fails is the first audio the next connection sends, but the offset was only anchored on push, so that frame kept the old connection's offset. The stream now anchors on the first frame each attempt consumes, with the capture time recorded at push. A node that builds events itself, relative to its first frame, was anchored on the pipeline's creation. An undeclared offset now means the stream's timeline starts with its first audio, so such streams keep a zero offset and the pipeline is stamped by the first frame pushed, as before this change. Only the default node, which declares a session-relative offset, is anchored on the session input start.
A resampler can hold audio across pushes, so its first output can carry samples from an earlier frame. Stamping it with the push that released it anchored the offset late by the time between the two pushes. Track the capture time of the oldest held sample and stamp each resampled frame from there.
A resampler can release audio from several pushes at once. Stamping the later outputs by adding durations erased the gap between those pushes, so a retry starting on one of them was anchored early. Track the held input as capture spans and advance through them per output frame.
Carry provider speech-end timing through END_OF_SPEECH so STT turn metrics retain the word timestamp instead of replacing it with event arrival. Keep unknown timestamps unset and clear Deepgram's cached final timing between turns and connections. Addresses AGT-3678 and the EOS reporting case in #7651. Co-authored-by: AndraΕΎ Pajtler <1963904+AndrazP@users.noreply.github.com>
chenghao-mou
added
plugins
review_effort:high
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Keep the change small. The offset is anchored on the first frame pushed to each attempt; a frame still queued when a connection fails is left as a TODO. Resampled frames are no longer stamped individually, so resampler buffering is a millisecond-level approximation.
Apply speaker filtering and formatting only to transcripts so timed EOS events still close turns and do not rewrite shared transcript text. Use word timing for Inference endpoints; billed audio duration does not identify when speech ended.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Soniox and Speechmatics Linden gateway finals carry utterance spans without word timestamps. Preserve those endpoints when every configured model uses that duration contract. Leave billing and unknown durations untimed. The timing semantics were checked against agent-gateway 69bcc23d088b.
Carry provider speech end timestamps and local VAD anchors through endpoint events. Normalize stream offsets and exclude billing duration, trailing silence, and audio from earlier pooled sessions. Add provider-specific regression coverage. Addresses AGT-3678 Co-authored-by: AndraΕΎ Pajtler <1963904+AndrazP@users.noreply.github.com>
The stream encoded one timeline in six fields and turned the caller's origin into an offset and back. Move the timeline into RecognizeStream._InputChan, which sees every push and every read. The channel stores the origin and anchors the offset when a connection reads its first frame. That frame was captured at the last push minus the audio still queued. This fixes the case where a connection fails before it reads: the retry now anchors on the oldest queued frame, not on the next push. A value set through the setter now means that the timeline began that many seconds ago, also after the first frame.
The push-audio stub left input_started_at as a truthy MagicMock, so no test reached the stamping branch. Reset it to None and assert it again. State the pipeline anchor contract next to the field.
The anchor took the last push minus the queued audio. A pause between queued frames is not in that sum, so the anchor moved forward by the pause. Also keep the push time of each queued frame. A frame is pushed after it is captured, so both values are upper bounds on its capture start: take the earlier one. It is exact across a pause and for a burst of buffered audio.
A leg was seeded only when it was created. If the caller set the offset after the first leg started, that leg kept a zero offset. Seed a leg that was not seeded when the first frame is forwarded to it. The outer stream has read that frame, so its timeline has a start.
The module docstring said a custom node is anchored on the pipeline's creation. It counts from its first audio, as on main.
Keep the ElevenLabs endpoint and reconnect fixes in this PR. Move the remaining provider changes to a separate follow-up so they can be reviewed independently.
Carry provider speech-end timing through endpoint events and re-anchor provider clocks after internal reconnects. Preserve queued and replayed audio timing without assigning speech-end times to untimed providers. Addresses AGT-3678 Co-authored-by: AndraΕΎ Pajtler <1963904+AndrazP@users.noreply.github.com> Co-authored-by: Long Chen <longch1024@gmail.com>
chenghao-mou
changed the title
chenghao-mou
removed
the
review_effort:high
chenghao-mou
changed the title
chenghao-mou
added
the
review_effort:high
There was a problem hiding this comment.
Note
Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.
Devin Review found 3 new potential issues.
1 flag not posted on this PR by your GitHub settings β view it in Devin Review. (Configure)
| alternatives=[ | ||
| stt.SpeechData( | ||
| language=LanguageCode(self._opts.language), | ||
| text="", | ||
| end_time=self._final_speech_end_time, | ||
| ) | ||
| ] | ||
| if self._final_speech_end_time > 0 | ||
| else [], |
There was a problem hiding this comment.
π‘ Sarvam endpoints lose late final timestamps
When vad.speech_end precedes transcript.final, _emit_end_of_speech sends an untimed endpoint before the final timestamp arrives. Manual endpointing also closes its turn before receiving the final. Those speech boundaries retain arrival-time timing instead of provider timing.
Learn more
The realtime stream sends END_OF_SPEECH directly from _handle_speech_end. Its provider timestamp is stored only when a transcript.final arrives. In manual mode, _end_manual_utterance also emits the endpoint immediately upon flushing. Once the endpoint has been sent, a later final cannot update its timing.
Example: The provider sends vad.speech_end, then transcript.final with end_s=0.4. The endpoint contains no alternatives even though the provider supplied a speech boundary of 0.4 seconds.
Recommended fix: Coordinate final and endpoint emission when provider timing is required, retaining the boundary until the final arrives or a bounded fallback expires. Preserve the existing ordering and terminal behavior when the final never arrives.
Was this helpful? React with π or π to provide feedback.
| alternatives=[self._speech_data(event, segments, text)] | ||
| if segments and float(segments[-1].get("end_time", 0)) > 0 | ||
| else [], |
There was a problem hiding this comment.
π‘ Baseten empty finals discard speech timing
When Baseten sends timed segments before an empty final, segments is empty at the endpoint. The endpoint loses the preceding segment's end time and falls back to arrival timing.
Learn more
The receiver keeps speaking across non-final messages, but it only reads segments from the current message. An empty commit-triggered final can therefore close a turn after an earlier timed transcription without carrying that earlier timestamp. The new endpoint alternatives stay empty.
Example: A non-final transcription supplies a segment ending at 0.4 seconds; the subsequent final has segments=[]. END_OF_SPEECH has no timestamp, although the preceding message supplied one.
Recommended fix: Keep the last valid segment end while the turn is open and use it for an empty final. Clear it when the turn closes so a later turn cannot inherit it.
Was this helpful? React with π or π to provide feedback.
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
1 flag not posted on this PR by your GitHub settings β view it in Devin Review. (Configure)
| if words and words[-1].get("end", 0) > 0: | ||
| self._final_alternatives = [speech_data] |
There was a problem hiding this comment.
π‘ New words inherit an old speech boundary
When wordless transcript.done adds words beyond the timed partial, _final_alternatives retains the partial's end time. The EOS event then places the new words before they were spoken, skewing turn timing.
Learn more
A timed partial can supply the only known word boundary for an utterance. A subsequent transcript.done with no words preserves that boundary here, even when its text adds speech beyond the partial. The EOS event passes that old time to AudioRecognition, where it becomes the turn's last speaking time. This is safe only when the wordless final repeats or summarizes speech already timed by partials.
Example: A partial reports hello ending at 0.4 seconds. A wordless transcript.done reports hello goodbye, with goodbye spoken later. EOS still reports 0.4 seconds as the end of all speech.
Recommended fix: Track the text covered by timed partials. Preserve the partial's boundary only when a wordless final contains no newly recognized speech; otherwise emit an untimed EOS or use a verified provider endpoint timestamp.
Was this helpful? React with π or π to provide feedback.
This branch has not been deployed
1 participant
Streaming STT plugins and hosted inference now attach available speech-end timing to EOS events, so turn metrics use the speech boundary. Internal reconnects re-anchor provider clocks while preserving queued and replayed audio timing. Providers without verified speech-end timing remain untimed.
Stacked on #7795. Addresses AGT-3678 (Linear) and #7651 (@AndrazP's EOS diagnosis). Reconnect handling follows @longcw's review.
Initial prompt and agent context
Model: GPT-6 Astra