Add Qwen3VL MCore Export support from PR 895

[hychiang-git]

This PR is duplicated from PR #895. Since the original branch source is not available now, we create a new branch where we can update this PR.

What does this PR do?

new feature:

Overview: Add Qwen3-VL (Vision-Language) model support to the Megatron Core export/import
plugin, enabling HuggingFace-to-mcore weight conversion for PTQ/QAT/QAD workflows

Details

Qwen3-VL has a different weight structure from Qwen3 text-only models:

• Language model weights are under model.language_model. prefix (not model.)
• Visual encoder weights are under model.visual. prefix
• The lm_head is at root level, not nested under language_model

This PR adds:

• mcore_qwen3vl.py: Import/export weight mapping rules between HuggingFace
  Qwen3VLForConditionalGeneration and Megatron Core, handling the language_model prefix for
  all decoder layers, QKV merging/slicing, gated MLP merging/slicing, Q/K layer norms.
• mcore_common.py: Registers Qwen3VLForConditionalGeneration in
  all_mcore_hf_export_mapping and all_mcore_hf_import_mapping.

Usage

From the comment:

Qwen3VL is supported within Megatron-Bridge, and pretraining and PEFT recipes for Qwen3VL are here and the core code logic here.

Create Megatron-LM/examples/post_training/modelopt/conf/Qwen/Qwen3-VL-8B-Instruct.sh :

#!/bin/bash
# Qwen3-VL-8B-Instruct text-model config for Megatron-LM import/quantize.
#
# Text-model dimensions are identical to Qwen3-8B (4096 hidden, 36 layers,
# 32 heads, GQA=8).  Differences: rope_theta=5000000, checkpoint path uses
# model.language_model.* prefix (handled by mcore_qwen3vl plugin).

if [ -z ${HF_MODEL_CKPT} ]; then
    HF_MODEL_CKPT=Qwen/Qwen3-VL-8B-Instruct
    TOKENIZER_MODEL=Qwen/Qwen3-VL-8B-Instruct
else
    TOKENIZER_MODEL=${HF_MODEL_CKPT}
fi

MODEL_ARGS=" \
    --save-interval 100000 \
    --micro-batch-size 1 \
    --bf16 \
    --no-masked-softmax-fusion \
    --disable-bias-linear \
    --untie-embeddings-and-output-weights \
    --position-embedding-type rope \
    --no-rope-fusion \
    --normalization RMSNorm \
    --swiglu \
    --num-layers 36 \
    --hidden-size 4096 \
    --ffn-hidden-size 12288 \
    --num-attention-heads 32 \
    --group-query-attention \
    --num-query-groups 8 \
    --kv-channels 128 \
    --qk-layernorm \
    --seq-length 4096 \
    --max-position-embeddings 262144 \
    --tokenizer-type HuggingFaceTokenizer \
    --make-vocab-size-divisible-by 1187 \
    --use-mcore-models \
    --rotary-percent 1.0 \
    --rotary-base 5000000 \
    --no-bias-swiglu-fusion \
"

Then, import Qwen3-VL from HuggingFace to MCore:

• Directly on a machine with GPUs (local):

MLM_MODEL_CFG=Qwen/Qwen3-VL-8B-Instruct \                                                                                                                                                                                                                                                                                                                                      
HF_MODEL_CKPT=Qwen/Qwen3-VL-8B-Instruct \                                                                                                                                                                                                                                                                                                                                      
MLM_MODEL_SAVE=/tmp/qwen3vl_mcore \                                                                                                                                                                                                                                                                                                                                            
TP=1 \                                                                                                                                                                                                                                                                                                                                                                         
bash Megatron-LM/examples/post_training/modelopt/convert.sh Qwen/Qwen3-VL-8B-Instruct                                                                                                                                                                                                                                                                                  

• To use it for quantize (PTQ via Megatron-LM path):

MLM_MODEL_CFG=Qwen/Qwen3-VL-8B-Instruct \                                                                                                                                                                                                                                                                                                                                      
HF_MODEL_CKPT=Qwen/Qwen3-VL-8B-Instruct \                                                                                                                                                                                                                                                                                                                            
QUANT_CFG=NVFP4_DEFAULT_CFG \                                                                                                                                                                                                                                                                                                                                                  
TP=4 \                                                                                                                                                                                                                                                                                                                                                                         
bash Megatron-LM/examples/post_training/modelopt/quantize.sh Qwen/Qwen3-VL-8B-Instruct 

Testing

• Verified round-trip import/export with Qwen3-VL-8B-Instruct with the example usage above
• Unit tests in tests/unit/torch/export/test_mcore_qwen3vl.py
  covering:
  • Registration in global export/import mappings
  • Import mapping: dense keys, model.language_model.
    prefix, lm_head. at root, QKVMerging, GatedMLPMerging, REPLICATE
    for layernorms, TP sharding configs
  • Export mapping: QKVSlicing, GatedMLPSlicing, no
    parallel_config
  • Import/export symmetry: same mcore keys, matching HF
    prefixes
  • Qwen3-VL vs Qwen3 difference: same keys, VL adds
    language_model. prefix, lm_head unchanged

Before your PR is "Ready for review"

• Is this change backward compatible?: Yes, additive only
• Did you write any new necessary tests?: Yes, tests/gpu_megatron/torch/export/test_mcore_qwen3vl.py
• Did you add or update any necessary documentation? Yes, see docs/source/deployment/3_unified_hf.rst
• Did you update Changelog? Yes, see CHANGELOG.rst

Additional Information

Companion Megatron-LM PR adds Qwen3VLModel, Qwen3VLDataset, and pretrain_qwenvl.py. Please see this PR NVIDIA/Megatron-LM#3444

Summary by CodeRabbit

• New Features

  • Added Megatron Core export/import mapping support for Qwen3‑VL, including handling of language‑model weight prefixes and support for both dense and MoE variants.

• Documentation

  • Added Qwen3‑VL (FP8 / NVFP4) to the deployment support matrix for TensorRT‑LLM.

• Tests

  • Added GPU tests validating registration, import/export mapping symmetry, and expected layer transformation behavior for Qwen3‑VL.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds bidirectional Megatron Core ↔ Hugging Face weight mappings for Qwen3-VL, registers them as a plugin, includes tests validating import/export symmetry and prefix rules, and updates changelog and deployment docs for Qwen 3‑VL support.

Changes

Qwen3-VL Megatron Core Integration

Layer / File(s)	Summary
Mapping Module Definition modelopt/torch/export/plugins/mcore_qwen3vl.py	New module introduces qwen3vl_causal_lm_import and qwen3vl_causal_lm_export dictionaries describing HF↔Megatron Core weight conversion, including prefix adjustments (model.language_model. vs root lm_head.), QKV/MLP merged-projection handling, and MoE expert slicing/routing.
Plugin Registration and Wiring modelopt/torch/export/plugins/mcore_common.py	Imports the new Qwen3-VL mapping functions and registers Qwen3VLForConditionalGeneration in both global export and import mapping dictionaries, wiring the model type to the new conversion handlers.
Test Suite and Validation tests/gpu_megatron/torch/export/test_mcore_qwen3vl.py	Comprehensive test suite with registration checks, key-presence validation for dense and MoE parameters, prefix behavior verification, transformation type checks for QKV/MLP, export-mapping func_kwargs checks, symmetry validation between import/export, and comparative assertions against Qwen3.
Documentation Updates CHANGELOG.rst, docs/source/deployment/3_unified_hf.rst	Changelog entry documenting Qwen3-VL Megatron Core support and deployment documentation noting TensorRT-LLM compatibility with FP8 and NVFP4 quantization formats.

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 5 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: adding Qwen3VL export/import support to Megatron Core, which aligns with the primary purpose of all file modifications.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Security Anti-Patterns	✅ Passed	All Python code changes pass security review. No unsafe deserialization, hardcoded trust_remote_code, eval/exec, # nosec bypasses, or new dependencies added. Code is declarative mappings only.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch hungyueh/pr-895

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@CHANGELOG.rst`:
- Line 136: Move the changelog entry "Add Megatron Core export/import mapping
for Qwen3-VL (``Qwen3VLForConditionalGeneration``) vision-language models..."
out of the 0.42 (2026-03-10) section and place it under the current
unreleased/0.45 section header in CHANGELOG.rst, preserving the existing
formatting and inline code markup; ensure you remove the duplicate from 0.42 and
verify the entry appears exactly once under the 0.45 (unreleased) section.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7f30ba6d-8de4-4386-b197-7f8189e56d24

📥 Commits

Reviewing files that changed from the base of the PR and between 62401e1 and 2423ae7.

📒 Files selected for processing (5)

• CHANGELOG.rst
• docs/source/deployment/3_unified_hf.rst
• modelopt/torch/export/plugins/mcore_common.py
• modelopt/torch/export/plugins/mcore_qwen3vl.py
• tests/gpu_megatron/torch/export/test_mcore_qwen3vl.py

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA.github.io/Model-Optimizer/pr-preview/pr-1482/
Built to branch gh-pages at 2026-05-14 20:12 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[hychiang-git]

/claude review

[claude]

Claude Review Summary

Small, additive PR that clones the Qwen3 mcore mapping with model. → model.language_model. substitution and registers Qwen3VLForConditionalGeneration. Mechanically straightforward; the risk is concentrated in a couple of places.

Findings

• CRITICAL: 0
• IMPORTANT: 2
• SUGGESTION: 1

Most impactful

1. MoE arch not registered (mcore_qwen3vl.py) — the file ships MoE rules (router, local_experts.*) and the comment claims support for "Qwen3-VL MoE variants like 30B-A3B", but only the dense Qwen3VLForConditionalGeneration arch is wired up in mcore_common.py. Either add the MoE arch entry (mirroring Qwen3MoeForCausalLM) or drop the MoE rules to avoid dead code that implies unsupported behavior.

2. lm_head placement — root-level lm_head. is inherited from mcore_qwen.py, but for several recent *ForConditionalGeneration VLMs in transformers, lm_head lives at model.language_model.lm_head.. If that holds for the Qwen3-VL release you target, import silently misses the tensor and export writes to the wrong key. PR description says round-trip was verified, so this may be fine — but worth a one-time safe_open(...).keys() confirmation.

3. Test placement — test_mcore_qwen3vl.py is dict-shape inspection only, doesn't need GPU or Megatron, but lives in tests/gpu_megatron/. Belongs in tests/unit/torch/export/ so it runs in the fast pre-merge lane (matches what the PR description said). Also note these tests assert "the dict we wrote equals the dict we wrote" — a small integration test against a real HF state-dict snapshot would catch the lm_head issue above.

Risk: Low-to-moderate. Code is purely additive, no existing arch behavior changes. Worst case is a broken Qwen3-VL round-trip that only manifests at runtime — which is exactly why the test placement matters.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.76%. Comparing base (a451a2b) to head (425145c).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1482      +/-   ##
==========================================
- Coverage   77.44%   76.76%   -0.68%     
==========================================
  Files         473      474       +1     
  Lines       51418    51445      +27     
==========================================
- Hits        39819    39493     -326     
- Misses      11599    11952     +353     

Flag	Coverage Δ
unit	52.55% <100.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hychiang-git]

/claude review

[claude]

Claude Review Summary

Findings: CRITICAL: 0 · IMPORTANT: 0 · SUGGESTION: 2

The change is small and additive: it registers a new HF→MCore mapping for Qwen3VLForConditionalGeneration that mirrors the existing Qwen3 mapping but threads the model.language_model. prefix used by the VL state dict (with lm_head. kept at root, matching the HF wrapper). No existing keys, defaults, or modelopt_state schema are touched, so backward-compatibility risk is essentially nil. The QKV/gated-MLP merging/slicing classes are reused, and dense + MoE keys are covered consistently in import and export.

Notes

• Test placement (suggestion): tests/gpu_megatron/torch/export/test_mcore_qwen3vl.py only inspects dict structure (no GPU, no Megatron, no checkpoint round-trip). It belongs in tests/unit/torch/export/ — that's also where the PR description says it lives. Moving it gets these structural checks into pre-merge CPU CI.
• Duplication (suggestion): mcore_qwen3vl.py is nearly a copy of mcore_qwen.py with model. → model.language_model.. A small derivation helper would keep them in lockstep when Qwen3 mappings evolve.

Risk

Low. The change is isolated to an additive plugin entry and a new file; the round-trip has been validated by the author against Qwen/Qwen3-VL-8B-Instruct. LGTM with the two suggestions above as follow-ups.

[claude]

Claude review passed — no significant issues found. LGTM (two non-blocking suggestions left inline).

[kevalmorabia97]

These are tests for very low level stuff. Can we instead just define a toy qwen3_vl HF model; import into Mcore and quantize with FP8 and export it so we can test this e2e?

You can define the toy model in https://github.com/NVIDIA/Model-Optimizer/blob/main/tests/_test_utils/torch/transformers_models.py
and refer to other megatron llm export tests how they are written for reference