Skip to content

fix(telemetry): never emit a user_speaking span that ends before it starts - #7263

Open
iamsrirams wants to merge 1 commit into
livekit:mainfrom
iamsrirams:fix/user-speaking-span-negative-duration
Open

iamsrirams wants to merge 1 commit into
livekit:mainfrom
iamsrirams:fix/user-speaking-span-negative-duration

Conversation

@iamsrirams

@iamsrirams iamsrirams commented Sep 14, 2026

Copy link
Copy Markdown

Fixes #3396.

The symptom

user_speaking spans occasionally report ~18446744073700 ms. That is a negative duration read as an unsigned 64-bit nanosecond count — the span ended before it started — and it swamps every other span on the trace. @longcw confirmed the diagnosis on the issue ("so the actual problem is the end_time < start_time"), and @Bnowako reported it again in March after a 1.3.9 → 1.3.12 bump.

Root cause

AudioRecognition._process_stt_event clamps the provider's speech_end_time to now, with an explicit comment saying why:

# clamped like the other anchors: a provider clock running ahead would
# otherwise push the anchor into the future and extend `extra_sleep`,
# delaying the turn commit by the skew
self._last_speaking_time = min(ev.speech_end_time, now)

The provider's speech_start_time, from the same event stream, is not clamped. That onset anchors both the user_turn and the user_speaking span, so a provider whose stream clock runs ahead starts them in the future while every end anchor is clamped to now — and the spans close before they opened.

It is reachable: SpeechEvent.speech_start_time is built as stream.start_time + provider_relative_offset (assemblyai, sarvam). stream.start_time is re-seeded on each retry, and audio pushed faster than realtime makes the provider's stream-relative offset outrun wall-clock.

The change

  1. voice/audio_recognition.py — clamp the provider onset to now, symmetric with the end anchor beside it. This is the root-cause fix.
  2. voice/agent_session.py — floor the user_speaking span's end at its own start, so no back-dated anchor from any source (a VAD silence window, a late provider timestamp) can invert it. This mirrors the floor _end_eou_wait_span already applies to eou_wait"resumed speech can carry a VAD timestamp from before the anchor; never negative" — and it keeps the back-dating, which fix user speaking span duration #3404 had to drop entirely (span.end() at wall-clock now) to avoid the same inversion.
  3. tests/trace_schema.py — teach the trace-shape checker the one invariant every span owes on its own: it never ends before it starts. The checker had rules for parentage and containment but none for a span's own bounds. It runs over the fake-session tests and over an OTLP export from a real run (python -m tests.trace_schema traces.json), so the next regression shows up in both.

user_speaking is a child of user_turn here, so change 1 also stops the turn span from starting in the future — which would otherwise put eou_wait (clamped to now) before its own parent.

Credit

@Bnowako proposed the same end-floor in #5160 back in March, along with the diagnosis on the issue. That PR predates the rewrite of agent_activity.py and no longer applies to this tree, so I have folded the floor in here rather than leaving it to rebase — full credit to them for it. Happy to drop change 2 and defer to a rebased #5160 if you would rather, though the clamp alone does not guarantee the invariant.

Verification

tests/test_user_speaking_span.py (new). Two of its four tests fail on unpatched main:

FAILED tests/test_user_speaking_span.py::test_provider_onset_ahead_of_the_local_clock_is_clamped
FAILED tests/test_user_speaking_span.py::test_user_speaking_span_never_ends_before_it_starts

E  AssertionError: user_speaking ends 500.0 ms before it starts;
   viewers read the negative duration as 18446744073210 ms

and pass with the change. The same before/after, driving the real _process_stt_event and the real _update_user_state end to end:

before:  provider onset  : now +0.500s     (stored as _speech_start_time)
         provider offset : now +0.000s     (stored as _last_speaking_time)
         span duration   : -499.84 ms  (as unsigned: 18446744073209.71 ms)

after:   provider onset  : now +0.000s
         span duration   : +0.14 ms

tests/test_trace_schema.py gets a self-test for the new checker rule, matching the file's existing style.

Full unit gate (pytest --unit) after the change: 2332 passed, with a failure set identical to the clean-tree baseline on this machine (the pre-existing failures there are missing provider deps, not related to this change). ruff check, ruff format --check and mypy -p livekit.agents.voice are clean.

@iamsrirams
iamsrirams requested a review from a team as a code owner September 14, 2026 03:56
@CLAassistant

CLAassistant commented Sep 14, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

…tarts

A `user_speaking` span whose end precedes its start is exported with a negative
duration, which viewers read as an unsigned 64-bit nanosecond count: the
~18446744073700 ms spans in livekit#3396 that swamp every other span on the trace.

Root cause: `_process_stt_event` clamps the provider's `speech_end_time` to
`now` — "a provider clock running ahead would otherwise push the anchor into the
future" — but does not clamp the provider's `speech_start_time` from the same
event stream. That onset anchors both the `user_turn` and the `user_speaking`
span, so a provider whose stream clock runs ahead (audio pushed faster than
realtime, a re-seeded stream anchor on retry) starts them in the future while
every end anchor is clamped. Clamp the onset the same way.

Also floor the span's end at its own start, so no back-dated anchor from any
source can invert it. This mirrors the floor `_end_eou_wait_span` already
applies to `eou_wait` ("resumed speech can carry a VAD timestamp from before the
anchor; never negative"), and keeps the back-dating that livekit#3404 had to drop to
avoid the same inversion.

Finally, teach the trace-shape checker the invariant every span owes on its own:
it never ends before it starts. The checker runs over the fake-session tests and
over an OTLP export from a real run, so the next regression shows up in both.
@iamsrirams
iamsrirams force-pushed the fix/user-speaking-span-negative-duration branch from e81047f to 41a6480 Compare September 14, 2026 03:57

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

user_speaking span sometimes has enormous duration_ms

2 participants