fix audio chop

[raghavm243512]

Fix choppy recorded audio in voice-agent benchmark runs

Problem

Under concurrency, recorded conversation audio (audio_user.wav / audio_mixed.wav) developed audible "chops" — hard-cut silence gaps in the middle of speech. It worsened with load and was most severe with the ElevenLabs user simulator. The live conversation and STT were unaffected (transcripts are coherent); the corruption was purely in the recording.

Root cause

The assistant servers reconstruct each channel's timeline by padding buffers with silence to match wall-clock time, read at frame-processing time (_pad_buffer_to_walltime in elevenlabs_server; pipecat 1.x's _fill_buffer_silence_gap; sync_buffer_to_position in the realtime/gemini servers). EVA's audio streams are already continuous (real audio + paced silence), so any wall-clock gap the recorder sees is event-loop jitter, not real silence. Under concurrency the loop stalls, frames are processed late and bunched, and the recorder inserts silence between frames whose audio is all present → a hard-cut chop.

This regressed recently via two changes that introduced wall-clock anchoring into the recording path: the pipecat 0.0.104 → 1.x upgrade (2026-06-01) and PR #154's per-frame _pad_buffer_to_walltime.

Fix (per backend, same principle: never fabricate silence from processing-time jitter)

• elevenlabs_server — pad the user track to wall-clock only at the start of a user turn, then append contiguously within the turn (mirrors the assistant pacer, which was already correct and never chopped).
• openai_realtime_server, gemini_live_server — guard the cross-track sync-pad: skip it when real user audio arrived within 300ms (the speaking-state flag goes stale under jitter). Safe-by-construction — only ever skips a pad, never adds silence.
• pipecat_server — ContiguousAudioBufferProcessor disables pipecat 1.x's wall-clock silence fill; byte-count track alignment is retained.

Diagnostic tooling

scripts/analyze_chops.py — detects chops (hard-cut digital-silence runs), separates LONG (≥150ms, audible) from short (~20ms single-frame, benign/pre-existing), and reports cross-channel overlap. Calibrated against labeled recordings (0 false positives on a clean file). Kept as a regression check; LONG chops is the metric that tracks audible quality.

Results

Account conc = ElevenLabs account load = concurrency × sides using ElevenLabs. "In-limit" ≈ ≤ ~30.

Config	Code	Acct conc	LONG chops	Overlap (s)	Verdict
ElevenLabs both	before	48 (over)	47	46.7	old code: choppy
ElevenLabs both	after	48 (over)	2	49.0	chops fixed; overlap unchanged (over-limit)
ElevenLabs both	after	28 (in)	0	3.9	✅ clean
ElevenLabs agent + realtime user	before	24	40	2.3	old code: choppy
ElevenLabs agent + realtime user	after	14	0	0.2	✅ clean
realtime agent + ElevenLabs user	after	14	3	38.1	chops fixed; overlap from ElevenLabs-as-user
gemini agent	before	24	3	0.0	old code: minor chops
gemini agent	after	24	0	2.8	✅ clean

Updated takeaways:

1. Chops fixed across all four backends: LONG chops drop to 0–3, and the fix holds even over the account limit (ElevenLabs both: 47 → 2 at acct 48).
2. No overlap regression: 46.7 → 49.0 at acct 48 (within noise). Gemini's 0.0 → 2.8 is negligible (~0.1s/record, within run-to-run variance).
3. Gemini was only marginally affected (3 LONG chops baseline vs. 40–47 for ElevenLabs/openai_realtime) — it shares the same sync_buffer_to_position pattern but was near-clean at baseline. The fix is validated (3 → 0) and consistent with the other backends; worth including so the known-vulnerable pattern isn't left in place.
4. Operating limit: cross-channel overlap degrades with ElevenLabs account concurrency past ~30 and with ElevenLabs-as-user burstiness — pre-existing, not introduced here, mitigated operationally (keep account concurrency within limits).

Takeaways:

1. Chops fixed across all backends: 40–47 LONG → 0–3, and the fix holds even over the account limit (47 → 2 at acct 48).
2. No overlap regression: 46.7 → 49.0 at acct 48 (within noise). Overlap is a pre-existing, separate issue.
3. New operating limit discovered: cross-channel overlap degrades when the ElevenLabs account is pushed past its concurrency limit (~30) and/or when ElevenLabs is the user sim (bursty delivery). It is not introduced by this change and is not fixable client-side; the mitigation is operational — keep account concurrency within limits (and prefer running metric-scored runs within them).