Repository navigation
Fix PCM16 and multichannel array input with NumPy CI - #3899
Merged
Merged
Conversation
load_audio_text_image_video averages the channels of audio files, but in-memory numpy arrays and tensors passed to generate() were handed on unchanged. A (1, N) array from torchaudio.load or a stereo array made the FSMN-VAD frontend assert on the window size and, with (N, 1), raise a ZeroDivisionError. Reduce 2-D floating-point waveforms to mono (the shorter axis is the channel axis) before resampling, honoring reduce_channels like the file path. Empty 2-D input becomes empty 1-D; integer input is left as before.
…_image_video A np.int16 array (for example from PyAudio or sounddevice) was converted to an int16 tensor without scaling. The frontend multiplies float waveforms by 32768, so the signal was overdriven and VAD segments changed, and with a sample rate other than 16 kHz the resampler raised a TypeError. Scale int16 arrays by 1/32768 like load_bytes does for raw PCM.
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Integrate #3890 and #3891, preserving both original contributor commits as ancestors and their implementation/tests unchanged. NumPy PCM16 sound inputs scale to float32; floating 2D sound arrays/tensors reduce to mono before resampling. Non-sound features, integer tensors and reduce_channels=False keep their existing behavior. The documented shorter-axis rule treats square inputs as channels-first.
Both contributor test files now trigger and execute in the existing NumPy 1.26.4 / 2.4.0 CI matrix, for pull requests and main pushes. No independent ONNX/libtorch package change.
Verification
Exact-head artifact acceptance
CUDA is unavailablecase fromtest_utterance_mvn.pyon the CPU runner. All 13 new input regression cases actually ran in both jobs.