fix(telemetry): never emit a user_speaking span that ends before it starts - #7263
Open
iamsrirams wants to merge 1 commit into
Open
iamsrirams wants to merge 1 commit into
iamsrirams wants to merge 1 commit into
Conversation
…tarts A `user_speaking` span whose end precedes its start is exported with a negative duration, which viewers read as an unsigned 64-bit nanosecond count: the ~18446744073700 ms spans in livekit#3396 that swamp every other span on the trace. Root cause: `_process_stt_event` clamps the provider's `speech_end_time` to `now` — "a provider clock running ahead would otherwise push the anchor into the future" — but does not clamp the provider's `speech_start_time` from the same event stream. That onset anchors both the `user_turn` and the `user_speaking` span, so a provider whose stream clock runs ahead (audio pushed faster than realtime, a re-seeded stream anchor on retry) starts them in the future while every end anchor is clamped. Clamp the onset the same way. Also floor the span's end at its own start, so no back-dated anchor from any source can invert it. This mirrors the floor `_end_eou_wait_span` already applies to `eou_wait` ("resumed speech can carry a VAD timestamp from before the anchor; never negative"), and keeps the back-dating that livekit#3404 had to drop to avoid the same inversion. Finally, teach the trace-shape checker the invariant every span owes on its own: it never ends before it starts. The checker runs over the fake-session tests and over an OTLP export from a real run, so the next regression shows up in both.
iamsrirams
force-pushed
the
fix/user-speaking-span-negative-duration
branch
from
September 14, 2026 03:57
e81047f to
41a6480
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3396.
The symptom
user_speakingspans occasionally report ~18446744073700ms. That is a negative duration read as an unsigned 64-bit nanosecond count — the span ended before it started — and it swamps every other span on the trace. @longcw confirmed the diagnosis on the issue ("so the actual problem is the end_time < start_time"), and @Bnowako reported it again in March after a 1.3.9 → 1.3.12 bump.Root cause
AudioRecognition._process_stt_eventclamps the provider'sspeech_end_timetonow, with an explicit comment saying why:The provider's
speech_start_time, from the same event stream, is not clamped. That onset anchors both theuser_turnand theuser_speakingspan, so a provider whose stream clock runs ahead starts them in the future while every end anchor is clamped tonow— and the spans close before they opened.It is reachable:
SpeechEvent.speech_start_timeis built asstream.start_time + provider_relative_offset(assemblyai, sarvam).stream.start_timeis re-seeded on each retry, and audio pushed faster than realtime makes the provider's stream-relative offset outrun wall-clock.The change
voice/audio_recognition.py— clamp the provider onset tonow, symmetric with the end anchor beside it. This is the root-cause fix.voice/agent_session.py— floor theuser_speakingspan's end at its own start, so no back-dated anchor from any source (a VAD silence window, a late provider timestamp) can invert it. This mirrors the floor_end_eou_wait_spanalready applies toeou_wait— "resumed speech can carry a VAD timestamp from before the anchor; never negative" — and it keeps the back-dating, which fix user speaking span duration #3404 had to drop entirely (span.end()at wall-clock now) to avoid the same inversion.tests/trace_schema.py— teach the trace-shape checker the one invariant every span owes on its own: it never ends before it starts. The checker had rules for parentage and containment but none for a span's own bounds. It runs over the fake-session tests and over an OTLP export from a real run (python -m tests.trace_schema traces.json), so the next regression shows up in both.user_speakingis a child ofuser_turnhere, so change 1 also stops the turn span from starting in the future — which would otherwise puteou_wait(clamped tonow) before its own parent.Credit
@Bnowako proposed the same end-floor in #5160 back in March, along with the diagnosis on the issue. That PR predates the rewrite of
agent_activity.pyand no longer applies to this tree, so I have folded the floor in here rather than leaving it to rebase — full credit to them for it. Happy to drop change 2 and defer to a rebased #5160 if you would rather, though the clamp alone does not guarantee the invariant.Verification
tests/test_user_speaking_span.py(new). Two of its four tests fail on unpatchedmain:and pass with the change. The same before/after, driving the real
_process_stt_eventand the real_update_user_stateend to end:tests/test_trace_schema.pygets a self-test for the new checker rule, matching the file's existing style.Full unit gate (
pytest --unit) after the change: 2332 passed, with a failure set identical to the clean-tree baseline on this machine (the pre-existing failures there are missing provider deps, not related to this change).ruff check,ruff format --checkandmypy -p livekit.agents.voiceare clean.