Repository navigation
fix(onnx): reset the frontend after a short final chunk in streaming Paraformer - #3927
Merged
LauraGPT merged 1 commit intoNov 20, 2026
Conversation
…Paraformer A final call to the streaming Paraformer with fewer than 960 samples decodes the overlap frames cached by the earlier calls and returns without calling extract_feat. extract_feat is what resets WavFrontendOnline at the end of a stream, so the frontend kept its cached samples, LFR splice frames and reserved waveform. The frontend belongs to the model, not to the caller's cache, so the next stream on the same model was framed as a continuation of the previous one: its first LFR frame was built from the end of the previous stream, and the 240-399 samples left in the frontend went in front of its audio, which also moved every later 25 ms window unless their number was a multiple of 160. This happens after any stream that has produced features and ends with an empty final call, and in the demo's 600 ms chunking whenever the audio ends with a chunk of 2 to 959 samples. Reset the frontend in that branch, as a final extract_feat call does. The C++ ParaformerOnline::Forward resets its frontend caches in the same branch, and the PyTorch model clears its frontend cache on is_final. The short final call decodes the same frames as before.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3926
funasr_onnx's streamingParaformer.__call__(runtime/python/onnxruntime/funasr_onnx/paraformer_online_bin.py:142-147) handles a final chunk with fewer than 960 samples, once the stream has produced features, by decoding the cached overlap frames and returning. It skipsextract_feat, and with it the frontend reset that every other final call gets:WavFrontendOnline.extract_fbankcallscache_reset()whenis_final(utils/frontend.py:377-378). Soinput_cache,lfr_splice_cacheandreserve_waveformssurvived the end of the stream.The frontend belongs to the model, not to the caller's
cache, so the next stream on the same model was framed as a continuation of the previous one: the 240-399 samples left ininput_cachewent in front of its audio, and its first LFR frame was built from the previous stream's last fbank frame(s). Its first 60 ms feature frame came from the end of the previous stream, and when the left-over sample count was not a multiple of 160, every later 25 ms window moved too. This happens after any stream that has produced features and ends with an empty final call, and with the 600 ms chunking ofdemo_paraformer_online.pywhenever the audio ends with a last chunk of 2 to 959 samples.This PR resets the frontend in that branch, as the final
extract_featcall does in every other case:The C++ runtime already does this in the same branch:
ParaformerOnline::ForwardcallsResetCache(), which clearsreserve_waveforms_,input_cache_andlfr_splice_cache_, thenReset()(runtime/onnxruntime/src/paraformer-online.cpp:532-539,:391-395). The PyTorch model keeps its frontend state incache["frontend"]and re-initializes the cache on every final call (funasr/models/paraformer_streaming/model.py:753-754,:547).The short final call itself decodes the same cached frames as before, and
inferdoes not read the frontend.The first encoder input of a 0.6 s stream decoded after another stream on the same model (the script in the issue):
mainType of change
Validation
The new
test_short_final_after_features_does_not_contaminate_next_streamgoes intests/test_funasr_onnx_paraformer_online_empty_final.py, next totest_short_final_does_not_contaminate_next_streamfrom #3827. That test covers a short first-and-final call, whose cache is still empty, so it goes throughextract_featand was already reset. The new test uses the class's model: real__call__,WavFrontendOnline, positions and overlap cache, withinfermocked. It decodes 9600 samples of a tone and then a final chunk of 0, 1, 500, 959 or 960 samples. It then decodes a 0.6 s stream of another tone on the same model and checks that the encoder input equals that of the same stream on a fresh model. 960 samples is the control that already goes throughextract_feat.Windows 11, CPU, Python 3.12.15, numpy 1.26.4, onnxruntime 1.32.0, kaldi-native-fbank 1.23.3, librosa 0.11.0. I built wheels from
runtime/python/onnxruntimeatmain(aa3d789) and at this branch, installed each into a fresh environment, and ran the test scripts astest-funasr-onnx-package.ymldoes:mainpackage:Ran 13 tests ... FAILED (failures=4). The subtests for 0, 1, 500 and 959 samples fail withArrays are not equal,Mismatched elements: 12 / 560,Max absolute difference: 14.8243885. The 960-sample subtest passes.Ran 13 tests ... OK.test_funasr_onnx_release.py(9 tests),test_funasr_onnx_installed.py(13),test_funasr_onnx_paraformer_online_overlap.py(3),test_funasr_onnx_vad_online_short_chunks.py(6),test_funasr_onnx_vad_multiple_inputs.py(6) andtest_funasr_onnx_punctuation_empty_input.py(5).With pytest over those seven files plus
test_rich_transcription_postprocess.py,test_rich_transcription_event_segments.py,test_runtime_cmvn.pyandtest_sentence_postprocess_timestamps.py(PYTHONUTF8=1):mainpackage:4 failed, 144 passed, 416 subtests passed. The 4 failures are the four subtests above.144 passed, 420 subtests passed. The test ids and outcomes are otherwise the same.Outside the test, with the 80-mel frontend of the issue's script and only
inferreplaced, I compared every encoder input betweenmainand this branch. The run had 60 models with 4 streams each, 0.6 to 2.5 s of a random tone plus noise per stream. The streams were cut three ways: 9600-sample chunks with the remainder as the final chunk, random chunks with an empty final call, or random chunks with a final chunk of 0-959 samples.On a fresh model, all 240 streams give identical encoder inputs and results on
mainand this branch.On a reused model, the 130 streams that came after a short final chunk differ from the same stream on a fresh model on
main(130/130), and match it on this branch (130/130). The 50 streams that came after a longer final chunk match on both.python -m compileall funasr examples testsDocs or links checked
Runtime/deployment command tested
black --line-length=100(24.4.0, the pre-commit pin) leaves the changed lines alone. It reports the same existing formatting in both files onmainand on this branch.git diff --checkis clean. No real model was run.User impact
Users of the
funasr_onnxstreaming Paraformer who decode several streams with one model, one after another, now get the features of each stream as on a fresh model. That covers a server that serves sessions in turn, or the demo looped over files. Before, after a stream that ended with an empty final call or a final chunk under 960 samples, the next stream started with the end of the previous one. Single streams, and streams that follow a final chunk of 960 samples or more, give the same encoder inputs as before.Notes for reviewers
infer, as it does on the other final path, whereextract_featresets the frontend before inference.test-funasr-onnx-package.ymlalready runs ("Verify empty split-final Paraformer result"). The package version stays at 0.4.11, which the release test checks.paraformer_online_bin.pyorutils/frontend.py.