Skip to content

ts_prediction_lfr6_standard drops a token timestamp when CIF must fire twice in one frame (ValueError in sentence_postprocess since 1.4.19) #3934

Closed

Description

@austery

🐛 Bug

ts_prediction_lfr6_standard can return fewer token timestamps than tokens when CIF needs to fire twice within a single frame. Since #3803 (1.4.19), sentence_postprocess correctly rejects that input with ValueError: time_stamp is missing a retained token timestamp. With AutoModel(..., vad_model=...), one such VAD segment fails the whole generate() call. Long recordings are lost because of one short segment.

We hit this with SEACO Paraformer on a 52-minute Mandarin recording. The failing VAD segment is about 120 ms long and holds a single token, 嗯.

Root cause

cif_wo_hidden checks integrate >= threshold once per frame, so each frame can fire at most once. In this segment the last frame alone carries alpha 1.72, and the accumulated weight goes from 0.28 to 2.00 in one frame. That is two fires' worth of weight, but only one fire is recorded:

alphas = [0.034, 0, 0, 0, 0.244, 1.723]   sum = 2.0
peaks  = [0.034, 0.034, 0.034, 0.034, 0.277, 2.000]
fires needed = len(char_list) + 1 = 2, fires found = 1

The renormalisation fallback (alphas /= alphas.sum() / (len(char_list) + 1)) does not help, because the sum is already 2.0. With one fire, range(len(fire_place) - 1) is empty, so the token gets no timestamp.

Check: keep the same sum but move weight out of the last frame ([0.034, 0, 0, 0, 0.9665, 1.0]). The function then returns [[50, 120]] and sentence_postprocess succeeds.

The same function and the same check are unchanged on current main and in 1.4.28. #3929 fixes a different path (symbol tokens in the mixed-text branch).

To Reproduce

  1. Install with: pip install funasr==1.4.19 (the code on the path is the same in 1.4.28 and on main)
  2. Run the code sample below.
  3. See the error: ValueError: time_stamp is missing a retained token timestamp

Code sample

import torch
from funasr.utils import postprocess_utils
from funasr.utils.timestamp_tools import ts_prediction_lfr6_standard

# CIF output captured from a real SEACO Paraformer VAD segment (~120 ms, one token).
us_alphas = torch.tensor([0.0335148, 0.0, 0.0, 0.0, 0.2435439, 1.7229414])
us_peaks = torch.tensor([0.0335148, 0.0335148, 0.0335148, 0.0335148, 0.2770587, 2.0])
token = ["嗯"]

_, timestamp = ts_prediction_lfr6_standard(us_alphas, us_peaks, list(token))
print("timestamp:", timestamp)  # [] -> one token, zero timestamps
postprocess_utils.sentence_postprocess(token, timestamp)  # ValueError

Expected behavior

Every retained token gets a timestamp, and short segments do not fail the whole request. Options include letting CIF fire more than once in a frame, or giving the missing tokens timestamps bounded by the segment in ts_prediction_lfr6_standard. Rejecting the input in sentence_postprocess is right; the producer should not create it.

For reference, our local workaround wraps ts_prediction_lfr6_standard in the seaco_paraformer model namespace. It keeps upstream timestamps and spreads the missing tokens over the rest of the segment. If upstream already stretched the last timed token to the segment end, it splits that token's span with them, which avoids zero-length timestamps.

Error logs

  File ".../funasr/auto/auto_model.py", line 893, in generate
    results = self.inference_with_vad(
  File ".../funasr/auto/auto_model.py", line 1115, in inference_with_vad
    results = self.inference(
  File ".../funasr/auto/auto_model.py", line 963, in inference
    res = model.inference(**batch, **kwargs)
  File ".../funasr/models/seaco_paraformer/model.py", line 565, in inference
    postprocess_utils.sentence_postprocess(token, timestamp)
  File ".../funasr/utils/postprocess_utils.py", line 196, in sentence_postprocess
    raise ValueError("time_stamp is missing a retained token timestamp")
ValueError: time_stamp is missing a retained token timestamp

Environment

  • OS: macOS 26.6 (arm64, Apple Silicon)
  • Python version: 3.11.14
  • FunASR version: 1.4.19 (path code identical in 1.4.28 and main)
  • ModelScope version: 1.32.0
  • PyTorch / torchaudio version: 2.13.0 / 2.11.0
  • Install method (pip, source, Docker): uv from PyPI
  • Device (cuda, cpu, mps): mps (the code sample also reproduces on CPU)
  • GPU model: Apple M4 Pro
  • CUDA/cuDNN version: n/a
  • Docker image tag, if used: n/a

Model: iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch with fsmn-vad (max_single_segment_time=30000), ct-punc, cam++; batch_size_s=60, merge_vad=True, merge_length_s=15.

Audio details

The audio cannot be shared.

  • Duration: 51.9 min recording; the failing VAD segment is about 120 ms
  • Sample rate: 16 kHz, mono
  • Format: MP3, 64 kbps
  • Language/dialect: Mandarin
  • Speaker count: 3
  • Background noise/music: none noted

Activity

  1. prassu017 commented on Nov 22, 2026

    @prassu017
    Contributor

    Reproduced on main (47ba463) with the captured alphas. #3935 spreads the missing timestamps over the rest of the segment instead of letting CIF fire twice in a frame, with your values as a test. @LauraGPT happy to be assigned if that works for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions