Skip to content

fix(websocket): pass the client's audio_fs to the Python server's models - #3937

Merged
LauraGPT merged 1 commit into
modelscope:mainfrom
tanishqbaweja:fix/wss-server-audio-fs
Nov 22, 2026
Merged

LauraGPT merged 1 commit into
modelscope:mainfrom
tanishqbaweja:fix/wss-server-audio-fs

Conversation

@tanishqbaweja

Copy link
Copy Markdown
Contributor

Summary

Fixes #3936

runtime/python/websocket/funasr_wss_server.py reads the client's audio_fs (:537-538 on main), but used it only to time the frames (:568) and for the WAV header of the --save_offline_segments files (:222). It gave the PCM to the VAD (:680), the streaming ASR (:823-829), the offline ASR (:736-742) and the speaker model (:703) without a sample rate. AutoModel reads PCM input at its fs argument, 16000 by default, and resamples it from there to the model's rate (funasr/auto/auto_model.py:863, funasr/utils/load_utils.py:199; the models read kwargs.get("fs", 16000), for example funasr/models/fsmn_vad_streaming/model.py:993). So audio sent with "audio_fs": 8000, which the protocol supports (runtime/docs/websocket_protocol.md:77) and funasr_wss_client.py sends for an 8 kHz WAV file (:253, :272), was read by all four models as 16 kHz audio, at twice its speed. The C++ servers resample each chunk from audio_fs to the model's rate (runtime/onnxruntime/src/audio.cpp:862, :883-885).

This PR passes audio_fs to those models as fs:

if "audio_fs" in messagejson:
    websocket.audio_fs = _safe_int(messagejson["audio_fs"], 16000)
    # AutoModel reads PCM input at fs (16000 by default) and resamples it
    # to the model's rate, as the C++ servers do with audio_fs.
    websocket.model_audio_fs = _model_audio_fs(websocket.audio_fs)
    websocket.status_dict_vad["fs"] = websocket.model_audio_fs
    websocket.status_dict_asr_online["fs"] = websocket.model_audio_fs
    websocket.status_dict_asr["fs"] = websocket.model_audio_fs

and _sv_and_match_sync gives the speaker model fs=websocket.model_audio_fs. The punctuation model takes text and is unchanged.

_model_audio_fs() passes on the standard rates 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100 and 48000 (MODEL_AUDIO_FS). Any other value keeps the models at 16000 Hz, as on main, and the server prints that the rate is not supported. AutoModel builds a torchaudio.transforms.Resample(audio_fs, fs) on every call (funasr/utils/load_utils.py:199-202), so passing the client's value as it is would let audio_fs of 0 or a negative rate make every model call fail, and a rate such as 44101 (co-prime with 16000) build a resampling kernel of several GB on every 60 ms VAD frame. From the standard rates the kernel is small. websocket.audio_fs itself is unchanged: it still times the frames and goes into the saved WAV header, as before.

Each connection also starts with fs 16000 in its three status dicts, so every call names its rate and a connection never reads its audio at another connection's rate through the shared models' settings.

The fs that each model's generate() received when the real funasr_wss_client.py --mode 2pass --audio_in <WAV file> streamed to the real server (stand-in models, see Validation):

WAV file main this PR
8 kHz ("audio_fs": 8000) none, so AutoModel reads 16000 8000 for VAD, streaming ASR, offline ASR and speaker model
16 kHz ("audio_fs": 16000) none, so AutoModel reads 16000 16000

Type of change

  • Bug fix
  • Documentation
  • Example or demo
  • Runtime or deployment
  • Benchmark or evaluation
  • Model/training change

Validation

The new tests/test_websocket_server_audio_fs.py loads the real server as tests/test_websocket_server_hotwords.py does, with AutoModel replaced, and runs ws_serve on 1.2 s of 2pass audio in 60 ms frames. Stand-in VAD, streaming ASR, offline ASR and speaker models record the fs that each generate() call receives (AutoModel reads 16000 when there is none):

  • test_models_read_pcm_at_the_client_sample_rate: audio_fs 8000, 16000 or not sent; every model reads the audio at 8000, 16000 and 16000.
  • test_wav_sample_rates_reach_the_models: 44100, 48000 and 22050, the rates of WAV files the bundled clients stream, reach every model.
  • test_unsupported_audio_fs_keeps_the_models_at_16000: 0, -8000, 44101 and "8k" keep every model at 16000, as on main.
  • test_every_model_call_names_its_sample_rate: without audio_fs, every call still passes fs.

tests/test_websocket_file_finalization.py runs ws_serve in its own namespace, so it now also provides _model_audio_fs there, next to its _safe_int.

Windows 11, CPU, pytest 9.2.0, on Python 3.12.15 (numpy 2.5.5, scipy 1.18.1, websockets 17.4), 3.11.17 (numpy 2.4.6, scipy 1.17.1, websockets 17.4) and 3.10.22 (numpy 2.2.6, scipy 1.15.3, websockets 16.1.1), the versions test-python-websocket.yml runs:

python -m pytest -q tests/test_websocket_server_asr_model.py tests/test_websocket_server_hotwords.py tests/test_websocket_file_finalization.py tests/test_websocket_client_hotwords.py tests/test_websocket_server_audio_fs.py
  • main (47ba463) server with these tests (Python 3.12): 5 failed, 46 passed. The 8000 case fails with {'vad': {16000}, 'online': {16000}, 'offline': {16000}, 'speaker': {16000}} where every model should read {8000}, the three WAV rates fail the same way, and the last test finds calls without fs. The unsupported values pass on main, which reads everything as 16000.
  • This branch: 51 passed on 3.12, 3.11 and 3.10.
  • The 40 tests of the four existing files pass with both servers.

I had no torch in this environment, so these tests ran with a stand-in torch module on PYTHONPATH that defines torch.Tensor, the only name the server uses; test-python-websocket.yml installs the real one.

I also ran the real funasr_wss_client.py --mode 2pass --audio_in <file> --send_without_sleep against the real server, both as processes on 127.0.0.1, with a stand-in funasr.AutoModel that writes down each generate() call (the script is in the issue). The client printed its final results each time. With a 2 s 8 kHz WAV file the client sent "audio_fs": 8000, and the VAD got 34 calls, the streaming ASR 4, the offline ASR 2 and the speaker model 2, with the same PCM on main and on this branch. On main none of these calls got an fs; on this branch all of them got fs=8000. With a 16 kHz WAV file, main gave no fs and this branch fs=16000. The punctuation model got the text and no fs with both servers.

  • python -m compileall funasr examples tests
  • Docs or links checked
  • Runtime/deployment command tested

black --line-length=100 (24.4.0, the pre-commit pin) leaves the new test unchanged and suggests nothing on the changed lines of funasr_wss_server.py and tests/test_websocket_file_finalization.py (it suggests the same older changes in both files on main). git diff --check is clean. No real model was run.

User impact

Users of the Python websocket server who send 8 kHz audio, such as telephone recordings through funasr_wss_client.py --audio_in, a WAV file in the HTML5 client, or any client that sets "audio_fs": 8000 as the protocol describes, now get it resampled to the models' 16 kHz, as with the C++ servers. Before, the models read it at twice its speed. Mono audio at the other standard rates that a client sends with audio_fs, such as a 44.1 or 48 kHz WAV file, is resampled too. Clients that send 16 kHz audio, with or without audio_fs, give the models the same input as before, and so do clients that send a value outside those rates.

Notes for reviewers

  • As in the C++ servers (Audio::LoadPcmwavOnline), the streaming models resample each piece of audio on its own: with the default options, a 60 ms frame for the VAD and a 600 ms chunk for the streaming ASR.
  • fs goes into the per-connection status dicts next to cache, is_final and chunk_size, and AutoModel.generate resets its runtime options before each call (_reset_runtime_configs). That reset and merge are not locked while several calls run on the shared models, so every connection now starts with fs 16000, and a connection that sends no audio_fs gives the models an explicit fs=16000, the rate they read before.
  • With an 8 kHz model (a frontend fs of 8000) as --asr_model or --vad_model, AutoModel on main resampled 8 kHz input from 16000 to 8000 Hz, to half its length; with this PR it reads it as it is.
  • test-python-websocket.yml now also runs tests/test_websocket_server_audio_fs.py (added to its path filters and to the pytest command).
  • No open PR touches runtime/python/websocket/.

The websocket protocol sends audio_fs with PCM audio, and its real-time
recognition takes 8000 and 16000 Hz. funasr_wss_client.py sends the
sample rate of its WAV file or its --audio_fs, and the HTML5 client that
of its WAV file. The C++ servers resample the audio to the model's rate.

funasr_wss_server.py used audio_fs only to time the frames, and passed
the PCM to AutoModel without it. AutoModel reads PCM input as 16 kHz
unless it is given fs, so the VAD, streaming ASR, offline ASR and
speaker models read 8 kHz audio as 16 kHz audio, at twice its speed.

Pass audio_fs to them as fs, the rate AutoModel resamples from, when it
is a standard rate (8000 to 48000). AutoModel builds a resampler from fs
on every call, so any other value (0, a negative or a rate such as
44101) keeps the models at 16000 Hz, as before. Each connection starts
with fs 16000, so every call names its rate. Clients that send 16000 or
no audio_fs give the models the same input as before.
@LauraGPT
LauraGPT merged commit 68aa96d into modelscope:main Nov 22, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

funasr_wss_server.py ignores the client's audio_fs: its models read 8 kHz PCM as 16 kHz audio

2 participants