Skip to content

fix: repair family parity and dependency handling - #1291

Draft
yifeif-nv wants to merge 8 commits into
NVIDIA:mainfrom
yifeif-nv:agent/nightly-single-gpu-repairs
Draft

yifeif-nv wants to merge 8 commits into
NVIDIA:mainfrom
yifeif-nv:agent/nightly-single-gpu-repairs

Conversation

@yifeif-nv

@yifeif-nv yifeif-nv commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

Background

Several declared single-GPU cases fail on numerical drift, missing tokenizer metadata, reference dependencies, or accumulated temporary bundles. These changes repair those paths while preserving the original numerical thresholds and frame integrity checks.

Exit Criteria

  • The affected original cases pass with unchanged inputs and numerical thresholds; frame order, count and completeness checks remain strict.
  • Current-main integration and required checks complete before merge.

Implementation

  • Accumulate LFM2 convolution products in FP32; use optimization level 0 for ELF's FP32 main engine, retaining level 1 for FP16.
  • Preserve Llama's inline chat template, load Qwen MoE's inline template, and normalize Nemotron Speech language-token metadata in a staged copy.
  • Declare PersonaPlex's reference dependencies and its pip build-isolation policy.
  • Release passed E2E cases' temporary directories before subsequent cases.
  • Complete Wan2.2 reference dependencies and accept the native/reference frame filenames while enforcing a continuous, ordered, homogeneous sequence.

Change categories

  • Model or runtime behavior
  • Dependencies
  • CI or developer tooling

Validation

Commands and Results

python -m tools.ci pipeline selective-e2e, using the original manifest selections:

Family Original cases Result
LFM2 lfm2-1.2b-greedy and five regression cases 6 passed
Llama nemotron-nano-4b, tinyllama-1.1b 2 passed on the current-main publishing candidate
Nemotron Speech nemotron-3.5-asr-streaming-0.6b Passed
PersonaPlex personaplex-7b, personaplex-7b-l0 2 passed
Qwen MoE qwen3-moe-30b-a3b, qwen3-moe-tiny-random 2 passed
ELF Original DE-EN, OWT and XSum L0 cases 3 passed
Wan2.2 wan22-ti2v-5b, wan22-ti2v-5b-l0 2 passed (official reference and original invariant-only L0)
Flux flux-2-dev, flux-2-dev-fp8 2 passed with the same controlled 240 GiB scratch limit

python3 -m tools.community_ci source-quality --base 74fe95229178d9e5980de30c024150a3a4680399: passed, including 172 tests on the publishing checkout.

Comparator regression plus existing reference/reporting tests: 36 passed. The same new regression tests against the old comparator reproduce six failures.

Hardware, Environment, and Revisions

Local Linux/aarch64 GPU validation used TensorRT 11.1.0.106 and Python 3.12, with the original checkpoint and precision selections. Publishing candidate: 135998f8936d17a66e6d38f3ce55e7401d0be688. Wan2.2 validation runs on this candidate; the other tested family implementations are byte-identical to their validated versions.

Not Run / Remaining Gaps

A complete final Source/tooling Nightly rerun has not completed; these local results do not establish full Nightly readiness.

Notes For Future Readers

Main already provides Llama's EOS support; this PR adds its template fallback. PersonaPlex needs tooling support for dependency-build.json before its opt-in is enabled.

One ELF qualification comparison measured about 35% higher sampling time and 25% higher decode time. This is a numerical-stability tradeoff, not a performance benchmark. Temporary cleanup retains call-failed directories, but not necessarily directories from later teardown/exit failures; exported evidence remains available.

Risk level

  • Medium

Several family outputs and dependency builds change; broader performance and full-suite qualification remain outstanding.

Round the completed depthwise convolution to the model dtype instead of rounding each product before reduction. The original six family E2E cases pass with their existing precision and acceptance criteria.

Signed-off-by: yifeif <277870278+yifeif-nv@users.noreply.github.com>
Copy the complete tokenizer_config chat template into the bundle when no standalone template exists. Preserve explicit-file precedence and reject unsupported metadata instead of silently using a raw prompt. Use the multi-EOS implementation already on main.

Signed-off-by: yifeif <277870278+yifeif-nv@users.noreply.github.com>
Move the checkpoint language-token list to additional_special_tokens in the existing staged tokenizer copy. Preserve token IDs, order, framing, and the original checkpoint files.

Signed-off-by: yifeif <277870278+yifeif-nv@users.noreply.github.com>
Declare einops and sphn for the official reference import closure. Select standard pip build isolation for the sphn source distribution through family-owned dependency metadata. Both original single-GPU cases pass using the dependency image built by the CI helper.

Signed-off-by: yifeif <277870278+yifeif-nv@users.noreply.github.com>
Fall back to the full tokenizer_config chat template when the bundle lacks a standalone template. Preserve raw-prompt behavior and reject unsupported metadata. The original 30B case now matches the reference tokens, and the tiny-random case also passes.

Signed-off-by: yifeif <277870278+yifeif-nv@users.noreply.github.com>
Use conservative builder optimization for iterative FP32 sampling while keeping FP16 policy unchanged. All three original family cases pass without changing acceptance criteria. This is a numerical mitigation; the observed sampling time increases about 35% in the paired qualification runs.

Signed-off-by: yifeif <277870278+yifeif-nv@users.noreply.github.com>
Remove passed-case temporary bundles before the next case builds. Preserve call-failed temporary directories and separately recorded evidence. The unchanged Flux pair reproduces ENOSPC before this change and passes afterward with the same 240 GiB private scratch limit.

Signed-off-by: yifeif <277870278+yifeif-nv@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Sep 14, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

Accept the native CLI and official reference fixed-width frame names while retaining strict sequence, file, count, and pixel checks. Add comparator regressions and declare the ARM64-compatible decord provider needed by the official reference. Existing E2E inputs and numeric thresholds stay unchanged.

Signed-off-by: yifeif <277870278+yifeif-nv@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant