Add MiniMax-H3

[apolinario]

Adds the MiniMax-H3 joint video and audio generation model: transformer, video VAE, audio VAE, scheduler, Modular Diffusers blocks for both released tasks, conversion script, docs, and tests.

Checkpoint: MiniMaxAI/MiniMax-H3 (single repo hosting both transformer variants at the root; the shared text encoder, VAEs and processor are stored once)

MiniMax-H3 generates video with synchronized stereo audio in a single denoising pass. One packed token sequence carries text, conditioning, audio and video rows through a shared 33B transformer; modality behavior comes from two input projections, a per row AdaLN modality tag, and two output heads. The released weights ship two transformers over shared components: one serving text to video+audio and first/last frame conditioning, one serving omni reference conditioning.

Modular only

The integration is Modular Diffusers blocks only, with no DiffusionPipeline half, the way Anima is. Two blocksets sit over one repository: MiniMaxH3Blocks (t2va, fl2va) reads transformer/, MiniMaxH3Ref2VABlocks (ref2va) reads transformer_ref/, and everything else is shared.

That shape is what makes one repository work. A modular index declares one loading spec per component, so from_pretrained fetches exactly the subfolders the blockset names and nothing else. It matters here beyond the two DiT partitions: the production repo hosts the original checkpoint folders alongside the converted ones, and a component wise index means loading either half never pulls the rest of the repo down.

Text to video and audio

import torch
from diffusers import ModularPipeline
from diffusers.utils import encode_video

pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

state = pipe(
    prompt="A red fox trotting through a snowy pine forest, snow crunching underfoot",
    num_frames=124,  # snapped up to 17n+5, 24 fps
    height=768,
    width=1344,
    num_inference_steps=30,
    generator=torch.Generator("cpu").manual_seed(42),
)
encode_video(
    state.get("videos")[0],
    fps=24,
    audio=state.get("audio")[0],
    audio_sample_rate=state.get("sampling_rate"),
    output_path="fox.mp4",
)

Video and audio come out of the one call as videos and audio, next to the sampling_rate of the soundtrack. Muxing them into one file is the caller's job.

First/last frame conditioning

image= conditions the first frame, last_image= the last, either or both (Wan style):

state = pipe(
    prompt="...",
    image=first_frame,        # PIL image, stretched to the target canvas
    last_image=last_frame,    # optional, cover cropped
    ...
)

Omni reference (image, video, audio references)

import torch
from diffusers.modular_pipelines import MiniMaxH3Ref2VABlocks
from diffusers.modular_pipelines.minimax_h3 import MiniMaxH3Reference

pipe = MiniMaxH3Ref2VABlocks().init_pipeline("MiniMaxAI/MiniMax-H3")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

state = pipe(
    prompt="The subject walks toward camera, matching the reference video's shot rhythm",
    references=[
        MiniMaxH3Reference(video="motion_ref.mp4"),  # motion and camera reference, soundtrack included
        MiniMaxH3Reference(image=subject_image),  # identity reference
        MiniMaxH3Reference(audio="voice.wav"),  # audio must be paired with an image or video reference
    ],
    num_frames=124,
    num_inference_steps=30,
)

A reference takes a path or a URL as well as in memory media, and decodes it as it is built, with PyAV for video and audio (an existing optional dependency, gated the way encode_video gates it). The rates come with it: a video reference reads its frame rate off the container and adopts its soundtrack when it has one, and an audio reference its sample rate. No block ever opens a file, and nothing is rebuilt: by the time a reference reaches the blocks, it is pixels and samples.

In memory media declares its own rates, defaulting to the model's own: fps=24.0 and the audio VAE sample rate, so only self generated data at another rate has to say so, and an explicit rate also wins over a container whose metadata is wrong. Frames land on the model's 24 fps grid by whole frame drop and duplicate, the same selection ffmpeg's fps filter made in the reference implementation, and a waveform is resampled once onto the audio VAE rate.

References pack in request order (order is load bearing for the model). Up to 9 images, 3 videos of 2 to 15 seconds, 3 audio clips, 12 references total. When exactly one audio bearing reference is present, num_frames may be omitted and the duration is that soundtrack's, snapped up to the next 17n+5; a soundtrack whose snapped duration leaves the 5 to 15 second window is rejected rather than silently stretched.

Two more shapes the same call covers. Multiple image references, where order assigns the roles the prompt names:

state = pipe(
    prompt="Use Image 1 for mood and visual language, use Image 2 as the protagonist reference",
    references=[
        MiniMaxH3Reference(image=style_image),
        MiniMaxH3Reference(image=protagonist_image),
    ],
    num_frames=124,
    num_inference_steps=30,
)

And speech synchronization from a single image plus a voice recording, where the duration comes from the audio and num_frames stays unset:

state = pipe(
    prompt="The character speaks in time with the reference recording, natural lip movement",
    references=[
        MiniMaxH3Reference(image=character_image),
        MiniMaxH3Reference(audio="voice.wav"),
    ],
    num_inference_steps=30,
)

Notes

• Guidance is distilled into the weights: no CFG, no negative_prompt, one forward per step.
• Two scheduler instances ride in the repo (scheduler/ for video at shift 12.0, audio_scheduler/ for audio at shift 3.0). MiniMaxH3Scheduler is a new class: the model predicts a data pointing velocity (x0 = x_t + sigma * v), which existing flow match schedulers cannot express.
• num_inference_steps counts sigma grid points, the terminal 0 included, so 30 steps drive 29 model evaluations and the progress bar shows 29.
• The video VAE decodes under fp16 autocast over fp32 weights, matching the reference recipe; tiling is on by default because the released model always tiles.
• 768p generation on a single 80GB card works with ComponentsManager.enable_auto_cpu_offload; the transformer alone is 61.7GB in bf16.
• pipe.transformer.set_attention_backend("_flash_3_hub") gives roughly 3x faster denoising on Hopper with kernels fetched from the Hub, no flash-attn build required.
• The ref2va entry point is MiniMaxH3Ref2VABlocks().init_pipeline(repo) today, and the design is ready for the planned ModularPipeline.from_pretrained workflow argument, at which point both halves load through from_pretrained directly.
• The modular tests read a tiny pipeline repo at hf-internal-testing/tiny-minimax-h3-modular-pipe, which does not exist yet; happy to hand over the builder script or the files for someone with access to create it.
• AutoencoderKLMiniMaxH3._encode_clip and ._encode carry @apply_forward_hook like the public encode/decode: keyframe and reference encoding goes through them rather than through encode, and without the hook the VAE stays on the CPU under offloading. The conditioner call fires the same hook by hand, because MiniMax-H3 reads hidden_states[50] off text_encoder.model and never uses the language model head.

Numerical parity against the reference implementation, verified per component (CPU float32) and end to end on GPU: the converted transformer reproduces the reference denoising trajectories bit for bit across all 15 documented use case configurations at 30 steps (both transformer variants, all aspect ratios, all conditioning modes).

[apolinario]

Self-review report

Pre-PR self-review per .ai/review-rules.md, updated for the modular-only design (this comment supersedes its earlier version, which reviewed the since-removed standard pipelines).

Design

• MiniMax-H3 ships as a modular-only integration (anima precedent): one repo, one modular_model_index.json, two blocksets. MiniMaxH3Blocks declares transformer, MiniMaxH3Ref2VABlocks declares transformer_ref; each half loads exactly its declared components. Verified empirically: a fresh-cache component load against the 210 GB repo fetched 578 MB, only the named subfolders, and the ref2va blockset never touches transformer/. This is what makes hosting the original checkpoint folders alongside the diffusers layout in one repo safe by construction.
• The ref2va entry point is MiniMaxH3Ref2VABlocks().init_pipeline(repo) today; both blocksets carry a _workflow_map so the planned ModularPipeline.from_pretrained(workflow=...) argument slots in without changes here.

Fixed during self-review and follow-up validation

• Offload-hook bypass class: submodule and private-method calls route around accelerate's CpuOffload forward wrapper (component silently stays on CPU). Three sites fixed: @apply_forward_hook on AutoencoderKLMiniMaxH3._encode_clip / ._encode, and both encode_prompts fire the hook manually before text_encoder.model(...) (routing through the top-level forward would run the 151936-way LM head purely to trigger a hook). The modular ComponentsManager offloading has the same semantics and is covered by the same fixes.
• Both VAEs pin themselves float32 under torch_dtype casts (_keep_in_fp32_modules, matching the transformer's mixed-precision contract): the released checkpoint is fp32 and a bf16 audio VAE decodes roughly 20 dB too quiet. The pin is asserted by positive tests rather than skips.
• Audio VAE attention refactored to the diffusers attention pattern (processor + AttentionModuleMixin + dispatch_attention_fn), verified bitwise against the reference on the real checkpoint. is_causal is honored by every registered backend except _native_npu (documented).
• Generator semantics are the standard convention: one generator, all draws via randn_tensor in a documented order, latents= injection, reproducibility pinned by tests on frames and audio (same seed twice identical, across t2va, keyframed and referenced runs).
• Duration ceiling validated on the aligned (17n+5) frame count everywhere, including the audio-derived-duration path.
• Docstring, dead-code and callback findings from the original pass: all applied, each gated by the corresponding bitwise parity suite.

Left for maintainer review, deliberately

• Weight-dtype-derived casts in the transformer and audio VAE: necessary under the mixed-precision checkpoint plus _keep_in_fp32_modules (where self.dtype is the wrong target for exactly the fp32 modules); interacts with quantized loading, so it needs an explicit ack rather than a silent pattern violation.
• _no_split_modules and the video VAE ViT decoder (register_tokens concat in forward): only a multi-GPU device_map="auto" run settles whether the decoder needs listing or repacking.
• Transformer attention_mask path is reachable only for hand-built padded layouts (both packers emit padless sequences); kept with test coverage, and the model docs note that padded layouts require a masked attention backend. Same mask-in-forward class as HunyuanVideo, but dormant in the shipped path.
• No MiniMaxH3LoraLoaderMixin in this PR, consistent with the no-LoRA-tests-initially convention.
• Full-graph torch.compile / torch.export are blocked by the data-dependent pad-mask branch; regional compilation (compile_repeated_blocks) and non-fullgraph compile work and are tested.

Verification summary: the transformer reproduces the reference implementation's denoising trajectories bit for bit across all 15 documented use-case configurations at 30 steps (both transformer partitions; verified against pinned reference commit 232f31f8b); VAEs, scheduler and packing verified bitwise (packing suites 68 + 211 checks); text-encoder tokens, tags and RoPE positions identical with embeddings at the attention-kernel noise floor; modular tests 69 passed (plus model tests, self-consistency and CPU smoke gates); ruff, check_copies, check_dummies, check_forward_call_docstrings, check_doc_toc, custom_init_isort, modular_auto_docstring all clean.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[perryyang-hue]

pipe.to("cuda"), out of memory for H100 84G memory

[yiyixuxu]

@apolinario is it ok we remove the code path related to padded-layout? our pipeline does not pad but there is a test suit for this so not sure if we can remove