Repository navigation
Whisper adapter - #1944
Open
dipo101 wants to merge 5 commits into
Open
Whisper adapter#1944dipo101 wants to merge 5 commits into
dipo101 wants to merge 5 commits into
Conversation
Audio encoder + text decoder models (Whisper) fit neither existing audio path: encoder-only audio models drop the tokenizer and feed `input` as input_values, and audio-text LMs load as AutoModelForSeq2SeqLM. - New AUDIO_SEQ2SEQ_ARCHITECTURES set, routed to AutoModelForSpeechSeq2Seq and classified as seq2seq. - Explicit cfg.is_audio_seq2seq flag. skip_tokenizer_for_modality keeps the tokenizer for it, and the processor loader runs AutoProcessor after the audio feature extractor so bridge.processor ends as the full processor (feature extractor + tokenizer). Audio-text LMs are unaffected. - forward(): the adapter declares the keyword audio goes in under (ArchitectureAdapter.audio_input_name); audio seq2seq models take their features as `input`, require decoder_input (or labels), and skip the shift-from-input_ids that text encoder-decoders use. 1-D decoder ids are batched. The input_values branch is untouched. - enable_compatibility_mode() refuses audio seq2seq with an accurate message until encoder weight processing lands. - n_ctx falls back to max_target_positions before the 2048 default.
So verify_models can run on Whisper: - build_modality_input builds the [batch, n_mels, 2 * n_frames] log-mel window for audio seq2seq. - The Phase 1 HF reference capture and benchmark_forward_pass pass the features under the adapter's audio_input_name plus decoder_start decoder ids, the same single-token convention text encoder-decoders use. - The component benchmark skips the channel-first front-end convs (covered end to end by the forward pass), and decoder_embed always gets token ids instead of the modality input. - The encoder-only Phase 8 audio suite skips audio seq2seq.
WhisperForConditionalGeneration (tiny through large-v3, large-v3-turbo,
distil-whisper) loads through boot_transformers with hooks in both the
audio encoder and the text decoder:
logits, cache = bridge.run_with_cache(input_features, decoder_input=ids)
WhisperArchitectureAdapter subclasses BartFamilyArchitectureAdapter
(pre-LN, per-stack final norms). What differs from the BART base:
- Encoder front end: conv1 / conv2 (channel-first, so their hooks declare
position on dim 2; hook_out is pre-GELU) and the fixed sinusoidal
embed_positions table, read whole as [n_frames, d_model], in place of
token embeddings. Two small seams on the base make this declarative:
_encoder_embeddings() and unembed_name.
- Unembedding is proj_out, tied to decoder_embed, with no
final_logits_bias.
- Stacks may be asymmetric (large-v3-turbo 32/4, distil-whisper n/2);
n_layers follows the decoder, as for Blenderbot.
- k_proj has no bias while q/v/out_proj do; noted in the docstring and
pinned by the tiny and whisper-tiny tests.
Out of scope here and planned as follow-ups: generate()
(supports_generation=False) and weight processing (compatibility mode
refuses). applicable_phases=[1] follows from that.
Registered in all four places. The model_type fallback and 17 registry
entries are added at status 0 (openai/whisper-*, distil-whisper/*).
Tests:
- unit: mapping, flags, routing, tokenizer carve-out, n_ctx, factory.
- test_whisper_tiny.py (CI, random weights, no download): independent HF
parity for logits and labels-only loss, hook shapes on both stacks,
front-end math, pos_slice on the declared axes, an encoder-output patch
that equals HF fed the patched encoder_outputs, error paths, and the
Phase 1 benchmarks.
- test_whisper_adapter.py (slow, openai/whisper-tiny): parity against an
independent eager HF model, the k_proj bias, and hook shapes.
Refs TransformerLensOrg#1938
verify_models on openai/whisper-tiny: P1 at 100% (all 74 components equivalent to HF, forward-pass logits within tolerance). P2-P4 are not applicable yet (applicable_phases=[1] until generate() and weight processing land). The entry, header counts and the verification_history record are what the tool wrote. The slow real-weights suite (test_whisper_adapter.py) passes 6/6, and teacher-forced logits match an independent eager HF model exactly (max abs diff 0.0 over all 51865 logits at each decoder position).
verify_models on openai/whisper-large-v3-turbo: P1 at 100% (all 242 components equivalent to HF, forward-pass logits within tolerance). This is the asymmetric case (32 encoder / 4 decoder layers) with the 128-mel-bin front end of the large-v3 family. The entry, header counts and the verification_history record are what the tool wrote.
dipo101
force-pushed
the
whisper-adapter
branch
from
October 23, 2026 23:10
9672083 to
26b528e
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
WhisperArchitectureAdapter, soWhisperForConditionalGenerationcheckpoints load viaboot_transformerswith hooks on both the audio encoder and the text decoder. This covers tiny through large-v3, large-v3-turbo and distil-whisper.conv1/conv2plus sinusoidalembed_positions, the unembed isproj_out, and asymmetric stacks follow the decoder.k_projhas no bias; this is documented and pinned in tests.AUDIO_SEQ2SEQ_ARCHITECTURESset (loads viaAutoModelForSpeechSeq2Seq);cfg.is_audio_seq2seqflag that keeps the tokenizer and loadsWhisperProcessor;audio_input_name="input_features", withdecoder_inputrequired.openai/whisper-tinyandopenai/whisper-large-v3-turbo(32/4 layers) are verified at P1 = 100%.Part of #1938.
generate()and weight processing follow in separate PRs; until then they raise clear errors.Type of change
Checklist: