Skip to content

Support packed multimodal inputs in Super hybrid models - #7253

Draft
tylerpoon wants to merge 3 commits into
NVIDIA:mainfrom
tylerpoon:tpoon/github-vlm-sft-06-super-hybrid-packed-inputs
Draft

Support packed multimodal inputs in Super hybrid models#7253
tylerpoon wants to merge 3 commits into
NVIDIA:mainfrom
tylerpoon:tpoon/github-vlm-sft-06-super-hybrid-packed-inputs

Conversation

@tylerpoon

Copy link
Copy Markdown
Contributor
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Derive Mamba sequence identifiers from packed boundaries, accept precomputed embeddings in hybrid MTP, and make packed MTP rolling handle empty shard-local documents. Include Mamba/MTP tests.

This is part of a review-split multimodal SFT series. Production run scripts, data YAMLs, checkpoint conversion tooling, and separate CI compatibility fixes are outside this series.

Dependencies and review scope

Targets main.

Validation

  • Replayed onto public main at 46fb90ca942a2950c58c5654b3e56877e88a3e35, preserving atomic signed and signed-off commits.
  • The original combined feature stack had multi-node Super SFT validation before this migration; that is not an isolated-PR or rebased-head test result.
  • GitHub CI is pending. No GPU tests have been rerun locally for these rebased heads.
  • Local Black, isort, pylint, and Ruff checks passed for paths covered by the repository autoformat script (megatron/core and tests). Mypy is unavailable locally. Changed Python files pass syntax parsing; no CI configuration changes are included.

Kept as a draft pending CI and dependency integration.

Signed-off-by: Tyler Poon <tpoon@nvidia.com>
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@tylerpoon

Copy link
Copy Markdown
Contributor Author

/ok to test f1c7937

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant