Repository navigation
fix(stt): anchor transcript timestamps on the session input start - #7795
Open
chenghao-mou wants to merge 15 commits into
Open
chenghao-mou wants to merge 15 commits into
chenghao-mou wants to merge 15 commits into
Conversation
chenghao-mou
added
the
review_effort:high
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
chenghao-mou
marked this pull request as draft
October 13, 2026 21:57
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
chenghao-mou
added this pull request to stack #7798
October 13, 2026 22:52
chenghao-mou
marked this pull request as ready for review
October 13, 2026 22:58
The default stt_node offsets provider timestamps from the session start, but the recognizer converted them with the pipeline's first audio frame. On a pipeline created after a handoff the two bases differ by the session's age, so stopped_speaking_at landed in the future and was clamped to the arrival time. The stream now absorbs the delay between its creation and its first frame, and re-anchors on the next frame after a reconnect, so its offset is exact on the caller's timeline. The default node records the base it measured from on the pipeline, and the recognizer adds that same base. A custom node that forwards provider timestamps untouched is anchored on the pipeline's creation, when it opens its stream. Addresses AGT-3678 Co-authored-by: Andraž Pajtler <1963904+AndrazP@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…irst-audio based Two review findings. A frame queued while a connection fails is the first audio the next connection sends, but the offset was only anchored on push, so that frame kept the old connection's offset. The stream now anchors on the first frame each attempt consumes, with the capture time recorded at push. A node that builds events itself, relative to its first frame, was anchored on the pipeline's creation. An undeclared offset now means the stream's timeline starts with its first audio, so such streams keep a zero offset and the pipeline is stamped by the first frame pushed, as before this change. Only the default node, which declares a session-relative offset, is anchored on the session input start.
A resampler can hold audio across pushes, so its first output can carry samples from an earlier frame. Stamping it with the push that released it anchored the offset late by the time between the two pushes. Track the capture time of the oldest held sample and stamp each resampled frame from there.
A resampler can release audio from several pushes at once. Stamping the later outputs by adding durations erased the gap between those pushes, so a retry starting on one of them was anchored early. Track the held input as capture spans and advance through them per output frame.
Keep the change small. The offset is anchored on the first frame pushed to each attempt; a frame still queued when a connection fails is left as a TODO. Resampled frames are no longer stamped individually, so resampler buffering is a millisecond-level approximation.
The stream encoded one timeline in six fields and turned the caller's origin into an offset and back. Move the timeline into RecognizeStream._InputChan, which sees every push and every read. The channel stores the origin and anchors the offset when a connection reads its first frame. That frame was captured at the last push minus the audio still queued. This fixes the case where a connection fails before it reads: the retry now anchors on the oldest queued frame, not on the next push. A value set through the setter now means that the timeline began that many seconds ago, also after the first frame.
The push-audio stub left input_started_at as a truthy MagicMock, so no test reached the stamping branch. Reset it to None and assert it again. State the pipeline anchor contract next to the field.
The anchor took the last push minus the queued audio. A pause between queued frames is not in that sum, so the anchor moved forward by the pause. Also keep the push time of each queued frame. A frame is pushed after it is captured, so both values are upper bounds on its capture start: take the earlier one. It is exact across a pause and for a burst of buffered audio.
A leg was seeded only when it was created. If the caller set the offset after the first leg started, that leg kept a zero offset. Seed a leg that was not seeded when the first frame is forwarded to it. The outer stream has read that frame, so its timeline has a start.
The module docstring said a custom node is anchored on the pipeline's creation. It counts from its first audio, as on main.
chenghao-mou
force-pushed
the
chenghao/fix/AGT-3678-stt-timestamp-anchor
branch
from
October 14, 2026 12:33
4e59347 to
3fd2c0e
Compare
longcw
reviewed
Oct 19, 2026
|
|
||
| def restart_provider_clock(self) -> None: | ||
| """Anchor the offset again on the next frame read, for a new connection.""" | ||
| self._first_frame_at = None |
Contributor
There was a problem hiding this comment.
if a plugin reconnect inside _run, e.g. for the update_options, it restarts the provider's clock but nothing calls restart_provider_clock, would this be a problem?
Member
Author
There was a problem hiding this comment.
good point. I am adding them in the #7796 so each plugin resets when it reconnects internally
longcw
approved these changes
Oct 21, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The default
stt_nodeoffsets provider timestamps from the session start, but the recognizer converted them with the pipeline's first audio frame. On a pipeline created after a handoff, the mismatch pushedstopped_speaking_atinto the future, and the clamp tonowreplaced it with the arrival time (cause 2 in #7651, AGT-3678).Streams now absorb the delay to their first frame and re-anchor after a reconnect, so
start_time_offsetis exact on the caller's timeline. The default node records its base on the pipeline and the recognizer adds that same base. Custom nodes keep their current behavior.Diagnosis and the first-frame approach come from @AndrazP's report. #7694 takes a broader approach to all three causes;
END_OF_SPEECHwithout a timestamp and mic gaps after the first frame remain open here.Addresses AGT-3678
Initial prompt and agent context
Model: Claude Fable 5.1
Later messages that changed the request: