Repository navigation
Description
🐛 Bug
ts_prediction_lfr6_standard can return fewer token timestamps than tokens when CIF needs to fire twice within a single frame. Since #3803 (1.4.19), sentence_postprocess correctly rejects that input with ValueError: time_stamp is missing a retained token timestamp. With AutoModel(..., vad_model=...), one such VAD segment fails the whole generate() call. Long recordings are lost because of one short segment.
We hit this with SEACO Paraformer on a 52-minute Mandarin recording. The failing VAD segment is about 120 ms long and holds a single token, 嗯.
Root cause
cif_wo_hidden checks integrate >= threshold once per frame, so each frame can fire at most once. In this segment the last frame alone carries alpha 1.72, and the accumulated weight goes from 0.28 to 2.00 in one frame. That is two fires' worth of weight, but only one fire is recorded:
alphas = [0.034, 0, 0, 0, 0.244, 1.723] sum = 2.0
peaks = [0.034, 0.034, 0.034, 0.034, 0.277, 2.000]
fires needed = len(char_list) + 1 = 2, fires found = 1
The renormalisation fallback (alphas /= alphas.sum() / (len(char_list) + 1)) does not help, because the sum is already 2.0. With one fire, range(len(fire_place) - 1) is empty, so the token gets no timestamp.
Check: keep the same sum but move weight out of the last frame ([0.034, 0, 0, 0, 0.9665, 1.0]). The function then returns [[50, 120]] and sentence_postprocess succeeds.
The same function and the same check are unchanged on current main and in 1.4.28. #3929 fixes a different path (symbol tokens in the mixed-text branch).
To Reproduce
- Install with:
pip install funasr==1.4.19(the code on the path is the same in 1.4.28 and onmain) - Run the code sample below.
- See the error:
ValueError: time_stamp is missing a retained token timestamp
Code sample
import torch
from funasr.utils import postprocess_utils
from funasr.utils.timestamp_tools import ts_prediction_lfr6_standard
# CIF output captured from a real SEACO Paraformer VAD segment (~120 ms, one token).
us_alphas = torch.tensor([0.0335148, 0.0, 0.0, 0.0, 0.2435439, 1.7229414])
us_peaks = torch.tensor([0.0335148, 0.0335148, 0.0335148, 0.0335148, 0.2770587, 2.0])
token = ["嗯"]
_, timestamp = ts_prediction_lfr6_standard(us_alphas, us_peaks, list(token))
print("timestamp:", timestamp) # [] -> one token, zero timestamps
postprocess_utils.sentence_postprocess(token, timestamp) # ValueErrorExpected behavior
Every retained token gets a timestamp, and short segments do not fail the whole request. Options include letting CIF fire more than once in a frame, or giving the missing tokens timestamps bounded by the segment in ts_prediction_lfr6_standard. Rejecting the input in sentence_postprocess is right; the producer should not create it.
For reference, our local workaround wraps ts_prediction_lfr6_standard in the seaco_paraformer model namespace. It keeps upstream timestamps and spreads the missing tokens over the rest of the segment. If upstream already stretched the last timed token to the segment end, it splits that token's span with them, which avoids zero-length timestamps.
Error logs
File ".../funasr/auto/auto_model.py", line 893, in generate
results = self.inference_with_vad(
File ".../funasr/auto/auto_model.py", line 1115, in inference_with_vad
results = self.inference(
File ".../funasr/auto/auto_model.py", line 963, in inference
res = model.inference(**batch, **kwargs)
File ".../funasr/models/seaco_paraformer/model.py", line 565, in inference
postprocess_utils.sentence_postprocess(token, timestamp)
File ".../funasr/utils/postprocess_utils.py", line 196, in sentence_postprocess
raise ValueError("time_stamp is missing a retained token timestamp")
ValueError: time_stamp is missing a retained token timestamp
Environment
- OS: macOS 26.6 (arm64, Apple Silicon)
- Python version: 3.11.14
- FunASR version: 1.4.19 (path code identical in 1.4.28 and
main) - ModelScope version: 1.32.0
- PyTorch / torchaudio version: 2.13.0 / 2.11.0
- Install method (
pip, source, Docker):uvfrom PyPI - Device (
cuda,cpu,mps):mps(the code sample also reproduces on CPU) - GPU model: Apple M4 Pro
- CUDA/cuDNN version: n/a
- Docker image tag, if used: n/a
Model: iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch with fsmn-vad (max_single_segment_time=30000), ct-punc, cam++; batch_size_s=60, merge_vad=True, merge_length_s=15.
Audio details
The audio cannot be shared.
- Duration: 51.9 min recording; the failing VAD segment is about 120 ms
- Sample rate: 16 kHz, mono
- Format: MP3, 64 kbps
- Language/dialect: Mandarin
- Speaker count: 3
- Background noise/music: none noted
Activity
fix(funasr): give filled timestamps at least one frame each
prassu017 commentedon Nov 22, 2026
Reproduced on main (47ba463) with the captured alphas. #3935 spreads the missing timestamps over the rest of the segment instead of letting CIF fire twice in a frame, with your values as a test. @LauraGPT happy to be assigned if that works for you.