Repository navigation
fix(websocket): pass the client's audio_fs to the Python server's models - #3937
Merged
Merged
Conversation
The websocket protocol sends audio_fs with PCM audio, and its real-time recognition takes 8000 and 16000 Hz. funasr_wss_client.py sends the sample rate of its WAV file or its --audio_fs, and the HTML5 client that of its WAV file. The C++ servers resample the audio to the model's rate. funasr_wss_server.py used audio_fs only to time the frames, and passed the PCM to AutoModel without it. AutoModel reads PCM input as 16 kHz unless it is given fs, so the VAD, streaming ASR, offline ASR and speaker models read 8 kHz audio as 16 kHz audio, at twice its speed. Pass audio_fs to them as fs, the rate AutoModel resamples from, when it is a standard rate (8000 to 48000). AutoModel builds a resampler from fs on every call, so any other value (0, a negative or a rate such as 44101) keeps the models at 16000 Hz, as before. Each connection starts with fs 16000, so every call names its rate. Clients that send 16000 or no audio_fs give the models the same input as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3936
runtime/python/websocket/funasr_wss_server.pyreads the client'saudio_fs(:537-538onmain), but used it only to time the frames (:568) and for the WAV header of the--save_offline_segmentsfiles (:222). It gave the PCM to the VAD (:680), the streaming ASR (:823-829), the offline ASR (:736-742) and the speaker model (:703) without a sample rate. AutoModel reads PCM input at itsfsargument, 16000 by default, and resamples it from there to the model's rate (funasr/auto/auto_model.py:863,funasr/utils/load_utils.py:199; the models readkwargs.get("fs", 16000), for examplefunasr/models/fsmn_vad_streaming/model.py:993). So audio sent with"audio_fs": 8000, which the protocol supports (runtime/docs/websocket_protocol.md:77) andfunasr_wss_client.pysends for an 8 kHz WAV file (:253,:272), was read by all four models as 16 kHz audio, at twice its speed. The C++ servers resample each chunk fromaudio_fsto the model's rate (runtime/onnxruntime/src/audio.cpp:862,:883-885).This PR passes
audio_fsto those models asfs:and
_sv_and_match_syncgives the speaker modelfs=websocket.model_audio_fs. The punctuation model takes text and is unchanged._model_audio_fs()passes on the standard rates 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100 and 48000 (MODEL_AUDIO_FS). Any other value keeps the models at 16000 Hz, as onmain, and the server prints that the rate is not supported. AutoModel builds atorchaudio.transforms.Resample(audio_fs, fs)on every call (funasr/utils/load_utils.py:199-202), so passing the client's value as it is would letaudio_fsof 0 or a negative rate make every model call fail, and a rate such as 44101 (co-prime with 16000) build a resampling kernel of several GB on every 60 ms VAD frame. From the standard rates the kernel is small.websocket.audio_fsitself is unchanged: it still times the frames and goes into the saved WAV header, as before.Each connection also starts with
fs16000 in its three status dicts, so every call names its rate and a connection never reads its audio at another connection's rate through the shared models' settings.The
fsthat each model'sgenerate()received when the realfunasr_wss_client.py --mode 2pass --audio_in <WAV file>streamed to the real server (stand-in models, see Validation):main"audio_fs": 8000)"audio_fs": 16000)Type of change
Validation
The new
tests/test_websocket_server_audio_fs.pyloads the real server astests/test_websocket_server_hotwords.pydoes, withAutoModelreplaced, and runsws_serveon 1.2 s of 2pass audio in 60 ms frames. Stand-in VAD, streaming ASR, offline ASR and speaker models record thefsthat eachgenerate()call receives (AutoModel reads 16000 when there is none):test_models_read_pcm_at_the_client_sample_rate:audio_fs8000, 16000 or not sent; every model reads the audio at 8000, 16000 and 16000.test_wav_sample_rates_reach_the_models: 44100, 48000 and 22050, the rates of WAV files the bundled clients stream, reach every model.test_unsupported_audio_fs_keeps_the_models_at_16000: 0, -8000, 44101 and"8k"keep every model at 16000, as onmain.test_every_model_call_names_its_sample_rate: withoutaudio_fs, every call still passesfs.tests/test_websocket_file_finalization.pyrunsws_servein its own namespace, so it now also provides_model_audio_fsthere, next to its_safe_int.Windows 11, CPU, pytest 9.2.0, on Python 3.12.15 (numpy 2.5.5, scipy 1.18.1, websockets 17.4), 3.11.17 (numpy 2.4.6, scipy 1.17.1, websockets 17.4) and 3.10.22 (numpy 2.2.6, scipy 1.15.3, websockets 16.1.1), the versions
test-python-websocket.ymlruns:main(47ba463) server with these tests (Python 3.12):5 failed, 46 passed. The 8000 case fails with{'vad': {16000}, 'online': {16000}, 'offline': {16000}, 'speaker': {16000}}where every model should read{8000}, the three WAV rates fail the same way, and the last test finds calls withoutfs. The unsupported values pass onmain, which reads everything as 16000.51 passedon 3.12, 3.11 and 3.10.I had no torch in this environment, so these tests ran with a stand-in
torchmodule onPYTHONPATHthat definestorch.Tensor, the only name the server uses;test-python-websocket.ymlinstalls the real one.I also ran the real
funasr_wss_client.py --mode 2pass --audio_in <file> --send_without_sleepagainst the real server, both as processes on 127.0.0.1, with a stand-infunasr.AutoModelthat writes down eachgenerate()call (the script is in the issue). The client printed its final results each time. With a 2 s 8 kHz WAV file the client sent"audio_fs": 8000, and the VAD got 34 calls, the streaming ASR 4, the offline ASR 2 and the speaker model 2, with the same PCM onmainand on this branch. Onmainnone of these calls got anfs; on this branch all of them gotfs=8000. With a 16 kHz WAV file,maingave nofsand this branchfs=16000. The punctuation model got the text and nofswith both servers.python -m compileall funasr examples testsblack --line-length=100(24.4.0, the pre-commit pin) leaves the new test unchanged and suggests nothing on the changed lines offunasr_wss_server.pyandtests/test_websocket_file_finalization.py(it suggests the same older changes in both files onmain).git diff --checkis clean. No real model was run.User impact
Users of the Python websocket server who send 8 kHz audio, such as telephone recordings through
funasr_wss_client.py --audio_in, a WAV file in the HTML5 client, or any client that sets"audio_fs": 8000as the protocol describes, now get it resampled to the models' 16 kHz, as with the C++ servers. Before, the models read it at twice its speed. Mono audio at the other standard rates that a client sends withaudio_fs, such as a 44.1 or 48 kHz WAV file, is resampled too. Clients that send 16 kHz audio, with or withoutaudio_fs, give the models the same input as before, and so do clients that send a value outside those rates.Notes for reviewers
Audio::LoadPcmwavOnline), the streaming models resample each piece of audio on its own: with the default options, a 60 ms frame for the VAD and a 600 ms chunk for the streaming ASR.fsgoes into the per-connection status dicts next tocache,is_finalandchunk_size, andAutoModel.generateresets its runtime options before each call (_reset_runtime_configs). That reset and merge are not locked while several calls run on the shared models, so every connection now starts withfs16000, and a connection that sends noaudio_fsgives the models an explicitfs=16000, the rate they read before.fsof 8000) as--asr_modelor--vad_model, AutoModel onmainresampled 8 kHz input from 16000 to 8000 Hz, to half its length; with this PR it reads it as it is.test-python-websocket.ymlnow also runstests/test_websocket_server_audio_fs.py(added to its path filters and to the pytest command).runtime/python/websocket/.