Skip to content

quantization.md

Latest commit

 

History

History
290 lines (220 loc) · 13.6 KB

quantization.md

File metadata and controls

290 lines (220 loc) · 13.6 KB
Raw

Model Quantization Techniques

📖 Overview

LightX2V supports quantized inference for DIT, VAE, T5, and CLIP models, reducing memory usage and improving inference speed by lowering model precision.

🔧 Quantization Modes

Quantization Mode Weight Quantization Activation Quantization Compute Kernel Supported Hardware
fp8-vllm FP8 channel symmetric FP8 channel dynamic symmetric VLLM H100/H200/H800, RTX 40 series, etc.
int8-vllm INT8 channel symmetric INT8 channel dynamic symmetric VLLM A100/A800, RTX 30/40 series, etc.
fp8-sgl FP8 channel symmetric FP8 channel dynamic symmetric SGL H100/H200/H800, RTX 40 series, etc.
fp8-f16-accum FP8 channel symmetric FP8 row-wise dynamic symmetric CUTLASS FP16 accumulation RTX 5090 (SM120)
int8-sgl INT8 channel symmetric INT8 channel dynamic symmetric SGL A100/A800, RTX 30/40 series, etc.
fp8-q8f FP8 channel symmetric FP8 channel dynamic symmetric Q8-Kernels RTX 40 series, L40S, etc.
int8-q8f INT8 channel symmetric INT8 channel dynamic symmetric Q8-Kernels RTX 40 series, L40S, etc.
int8-torchao INT8 channel symmetric INT8 channel dynamic symmetric TorchAO A100/A800, RTX 30/40 series, etc.
int4-g128-marlin INT4 group symmetric FP16 Marlin H200/H800/A100/A800, RTX 30/40 series, etc.
fp8-b128-deepgemm FP8 block symmetric FP8 group symmetric DeepGemm H100/H200/H800, RTX 40 series, etc.

🔧 Obtaining Quantized Models

Method 1: Download Pre-Quantized Models

Download pre-quantized models from LightX2V model repositories:

DIT Models

Download pre-quantized DIT models from Wan2.1-Distill-Models:

# Download DIT FP8 quantized model
huggingface-cli download lightx2v/Wan2.1-Distill-Models \
    --local-dir ./models \
    --include "wan2.1_i2v_720p_scaled_fp8_e4m3_lightx2v_4step.safetensors"

Encoder Models

Download pre-quantized T5 and CLIP models from Encoders-LightX2V:

# Download T5 FP8 quantized model
huggingface-cli download lightx2v/Encoders-Lightx2v \
    --local-dir ./models \
    --include "models_t5_umt5-xxl-enc-fp8.pth"

# Download CLIP FP8 quantized model
huggingface-cli download lightx2v/Encoders-Lightx2v \
    --local-dir ./models \
    --include "models_clip_open-clip-xlm-roberta-large-vit-huge-14-fp8.pth"

Method 2: Self-Quantize Models

For detailed quantization tool usage, refer to: Model Conversion Documentation

🚀 Using Quantized Models

DIT Model Quantization

Supported Quantization Modes

DIT quantization modes (dit_quant_scheme) support: fp8-vllm, int8-vllm, fp8-sgl, fp8-f16-accum, int8-sgl, fp8-q8f, int8-q8f, int8-torchao, int4-g128-marlin, fp8-b128-deepgemm

Configuration Example

{
    "dit_quantized": true,
    "dit_quant_scheme": "fp8-sgl",
    "dit_quantized_ckpt": "/path/to/dit_quantized_model"  // Optional
}

💡 Tip: When there's only one DIT model in the script's model_path, dit_quantized_ckpt doesn't need to be specified separately.

MiniMax-H3 FP8 with FP16 Accumulation

On RTX 5090, MiniMax-H3 can use FP8 inputs with FP16 accumulation through fp8-f16-accum. Convert the weights with the h3-fp8-f16-accum profile; regular fp8-sgl checkpoints are not compatible. DiT and Video VAE decoder are converted separately. The profile selects the qmax-14 projections, keeps standard FP8 quantization for the remaining layers, and records the policy in safetensors metadata.

python tools/convert/converter.py \
    --source /path/to/MiniMax-H3/transformer \
    --output /path/to/h3_quantized \
    --output_name minimax_h3_dit_fp8_f16_accum \
    --output_ext .safetensors \
    --model_type h3 \
    --device cuda \
    --quantized \
    --bits 8 \
    --linear_type fp8 \
    --quantization_profile h3-fp8-f16-accum \
    --single_file

python tools/convert/converter.py \
    --source /path/to/MiniMax-H3/vae \
    --output /path/to/h3_quantized \
    --output_name minimax_h3_video_vae_fp8_f16_accum \
    --output_ext .safetensors \
    --model_type h3_video_vae_decoder \
    --device cuda \
    --quantized \
    --bits 8 \
    --linear_type fp8 \
    --quantization_profile h3-fp8-f16-accum \
    --single_file
{
  "dit_quantized": true,
  "dit_quant_scheme": "fp8-f16-accum",
  "dit_quantized_ckpt": "/path/to/minimax_h3_dit_fp8_f16_accum.safetensors",
  "video_vae_quantized": true,
  "video_vae_quant_scheme": "fp8-f16-accum",
  "video_vae_quantized_ckpt": "/path/to/minimax_h3_video_vae_fp8_f16_accum.safetensors"
}

Activations use dynamic row-wise quantization with scale = max(abs(x)) / qmax. Reducing qmax increases the scale and lowers the raw values accumulated in FP16, at the cost of fewer effective FP8 levels. In the validated MiniMax-H3 workload, DiT produced non-finite FFN-out values with qmax 14 and 12, while qmax 7 completed every denoising step. Video VAE decoder remained finite and had the lowest error with qmax 14. The current H3 policy therefore fixes activation qmax to 7 for DiT and 14 for Video VAE, avoiding mismatches between runtime configuration and checkpoint conversion.

The kernel is enabled only for DiT Q/K/V, attention output, and FFN projections, and for Video VAE packed QKV, attention output, and FFN projections. It falls back to fp8-sgl when the extension is unavailable or the device is not SM120. DiT tensor parallel also currently uses the fp8-sgl fallback. Initialization logs report the effective scope or fallback reason. All other pipeline settings are independent of this quantization mode.

The kernel automatically tunes its CUTLASS tile and swizzle for each exact GEMM shape. The first use of an unseen shape benchmarks the built-in candidates in C++ and keeps the winner in a process-local C++ cache; later calls in the same process perform only a cache lookup. Enabling warmup moves the tuning cost out of the first request when warmup covers the production shapes. A restarted process tunes its shapes again and does not write to the user's cache directory.

MiniMax-H3 Video VAE Encoder Conv3D

The MiniMax-H3 Video VAE Encoder supports the default PyTorch path and three optional Conv3D modes:

Mode Implementation Weight qmax Activation qmax Accumulator Checkpoint constraint
torch (default) PyTorch Conv3D - - Backend-managed Original Encoder weights
torch_channels_last PyTorch Conv3D with channels_last_3d - - Backend-managed Original Encoder weights
cutlass_fp8_f32_accum CUTLASS FP8 Conv3D 448 448 FP32 Standard FP8 weight and scale
cutlass_fp8_f16_accum CUTLASS FP8 Conv3D 21 21 FP16 h3-vae-encoder-fp8-f16-accum profile

These modes affect only tasks that invoke the Video VAE Encoder, such as I2AV and REF2AV. T2AV does not execute the Encoder. The torch_channels_last mode uses the same PyTorch operators and dtype policy; this setting changes only the tensor layout.

FP32 accumulation uses the standard full E4M3 range and does not require a named profile. FP16 accumulation requires the checkpoint profile because weight and activation quantization must share its validated qmax constraint. It is the more aggressive mode and should be qualified for output quality before deployment.

The converter starts from a runtime-compatible combined H3 Video VAE FP8 checkpoint. Its Decoder Linear weights and scales must already be FP8 and FP32, with FP16 biases. In this source checkpoint, all Encoder tensors retain the original FP32 dtype. The converter validates but does not modify the Decoder. It quantizes 27 of the 33 Encoder Conv3D weights; the remaining six sensitive weights and all Encoder biases stay FP32. It does not convert a BF16 Decoder. Use the same Encoder FP8 mode for conversion and runtime execution. Decoder tensors and metadata are preserved, so video_vae_quant_scheme must match the source Decoder (fp8-sgl or fp8-f16-accum). The example below starts from an FP8-F16 Decoder checkpoint meeting the dtype requirements above and adds Encoder qmax-21 quantization; it does not change the Decoder's qmax-14 policy:

python tools/convert/converter.py \
    --source /path/to/minimax_h3_video_vae_fp8_f16_accum.safetensors \
    --output /path/to/h3_quantized \
    --output_name minimax_h3_video_vae_fp8_f16_accum_encoder_conv_q21 \
    --output_ext .safetensors \
    --model_type h3_video_vae_encoder \
    --vae_encoder_conv_mode cutlass_fp8_f16_accum \
    --device cuda \
    --quantized \
    --bits 8 \
    --linear_type fp8 \
    --single_file

Add the matching mode and checkpoint to the complete startup JSON selected by --config_json, or by Python create_generator(config_json=...):

{
    "video_vae_quantized": true,
    "video_vae_quant_scheme": "fp8-f16-accum",
    "video_vae_quantized_ckpt": "/path/to/minimax_h3_video_vae_fp8_f16_accum_encoder_conv_q21.safetensors",
    "vae_encoder_conv_mode": "cutlass_fp8_f16_accum"
}

These settings are loaded once with the VAE and apply to subsequent requests. Keep model_variant as the startup weight selection (fl2av or ref2av); choose task within that variant in each request. Request overrides continue to use size=[height, width], num_frames, and save_result_path; fps, compile, warmup, and vae_encoder_conv_mode remain startup settings. The conversion command above is an offline weight-conversion tool, so its output arguments describe a checkpoint rather than an inference result.

FP8 Conv3D currently supports SM120 only and requires a lightx2v_kernel_nvidia (or compatible legacy lightx2v_kernel) wheel built from a revision containing the FP8 Conv3D operator. A new Conv3D shape is tuned automatically on first use and its winner is reused by later calls in the same process; no separate offline tuning step is required.

T5 Model Quantization

Supported Quantization Modes

T5 quantization modes (t5_quant_scheme) support: int8-vllm, fp8-sgl, int8-q8f, fp8-q8f, int8-torchao

Configuration Example

{
    "t5_quantized": true,
    "t5_quant_scheme": "fp8-sgl",
    "t5_quantized_ckpt": "/path/to/t5_quantized_model"  // Optional
}

💡 Tip: When a T5 quantized model exists in the script's specified model_path (such as models_t5_umt5-xxl-enc-fp8.pth or models_t5_umt5-xxl-enc-int8.pth), t5_quantized_ckpt doesn't need to be specified separately.

CLIP Model Quantization

Supported Quantization Modes

CLIP quantization modes (clip_quant_scheme) support: int8-vllm, fp8-sgl, int8-q8f, fp8-q8f, int8-torchao

Configuration Example

{
    "clip_quantized": true,
    "clip_quant_scheme": "fp8-sgl",
    "clip_quantized_ckpt": "/path/to/clip_quantized_model"  // Optional
}

💡 Tip: When a CLIP quantized model exists in the script's specified model_path (such as models_clip_open-clip-xlm-roberta-large-vit-huge-14-fp8.pth or models_clip_open-clip-xlm-roberta-large-vit-huge-14-int8.pth), clip_quantized_ckpt doesn't need to be specified separately.

Performance Optimization Strategy

If memory is insufficient, you can combine parameter offloading to further reduce memory usage. Refer to Parameter Offload Documentation:

📚 Related Resources

Configuration File Examples

Run Scripts

Tool Documentation

Model Repositories

Through this document, you should be able to:

✅ Understand quantization schemes supported by LightX2V ✅ Select appropriate quantization strategies based on hardware ✅ Correctly configure quantization parameters ✅ Obtain and use quantized models ✅ Optimize inference performance and memory usage

If you have other questions, feel free to ask in GitHub Issues.