Repository navigation
fix(postprocess): keep a timestamp for symbol tokens in sentence_postprocess - #3929
Merged
LauraGPT merged 1 commit intoNov 20, 2026
Merged
Conversation
…process Tokens such as "." and "&" fall into the last branch of the mixed-text path, which added the word but no timestamp (funasr) or raised ValueError (funasr_onnx, funasr_torch). Every word from that token on was paired with the next token's span and the last word got none, and acronyms after the symbol could raise IndexError. Give the token its own timestamp in all three copies, and handle the blank after a Latin word like the Chinese branch does.
Merged
LauraGPT
added a commit
that referenced
this pull request
Nov 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In
sentence_postprocess, a token that is not Chinese, a digit, a BPE piece or letters (the Paraformer-zh vocabulary has.and&) falls into the last branch of the mixed-text path. That branch adds the word but no timestamp infunasr/utils/postprocess_utils.py, and raisesValueError: invalid characterin thefunasr_onnxandfunasr_torchcopies. With timestamps requested, every word from that token on is paired with the next token's span and the last word gets none, and a spelled-out acronym after it can raiseIndexError:Same error as #1283 in
funasr_onnx(ValueError: invalid character: .). It reaches users throughAutoModel("paraformer-zh"):SeacoParaformer.inference(funasr/models/seaco_paraformer/model.py) callssentence_postprocess(token, timestamp)and returns the result asresult["timestamp"].The token now gets its own
[start, end]in all three copies, so words and timestamps stay one to one.Type of change
Validation
python -m compileall funasr examples testspython -m pytest tests/test_sentence_postprocess_timestamps.py -q: 14 failed subtests (3 new tests over the three copies), 13 passed on current main; 13 passed (57 subtests) with the fix.test_postprocess_hotwords.py,test_rich_transcription_postprocess.py,test_sentence_postprocess_timestamps.py): 44 passed, 6 skipped (fuzzy hotword tests, optional deps not installed; the same 6 skip on main).User impact
Word-level
timestampoutput for Paraformer-style models stays aligned with the words when the text contains a decimal point,&or similar tokens;funasr_onnxandfunasr_torchno longer raise on them.Notes for reviewers
["ok", ".", "好"]givesok.好without timestamps and keeps好at its own span with them.