test(qwen3_5): qualify the remaining published Qwen3.5 widths

[zhenshanx-nv]

Background

families/qwen3_5 claims every dense Qwen3.5 checkpoint by model_type, but only
Qwen/Qwen3.5-9B had a manifest. The three smaller published widths — 0.8B, 2B and 4B —
were therefore supported without ever being measured against the reference.

Exit Criteria

• Each published dense width builds and matches the Hugging Face reference.
• No change to the family, or the claim in support.py was wrong and that is a different PR.
• The premerge suite does not get slower.

Implementation

Manifests only. No family code changed.

Routing was checked first, because qwen3_5 and qwen3_8 both name model_type: qwen3_5
and a checkpoint claimed by two families is a hard failure in resolve_family. They are
exact complements by construction: families/qwen3_5/support.py returns None when
output_gate_type is present without mlp_only_layers, which is precisely the condition
families/qwen3_8/support.py claims. All three widths report no output_gate_type and do
carry mlp_only_layers, so they resolve to qwen3_5 alone.

Change categories

• CI or developer tooling

Validation

Commands and Results

Each width built at fp16 and driven through the native CLI, compared against the Hugging
Face reference with the prompt rendering and metric used by
families/qwen3_5/tests/test_e2e.py:

width	engine token ids	text	reference	NED
Qwen3.5-0.8B	[11751, 248046, 198, 248044]	Paris	identical	0.0000
Qwen3.5-2B	[57590, 248046, 198, 248044]	Paris	identical	0.0000
Qwen3.5-4B	[57590, 248046, 198, 248044]	Paris	identical	0.0000

Token for token, against a threshold of 0.15.

• tools/tests/test_architecture.py and apps/benchmark/trtmc_benchmark/tests — 154 passed
  (this covers the manifest field contract and the release-matrix coverage rule)

Hardware, Environment, and Revisions

• NVIDIA A100 80GB PCIe, x86 dev container
• Checkpoints: Qwen/Qwen3.5-0.8B, Qwen/Qwen3.5-2B, Qwen/Qwen3.5-4B
• Rebased on c96119fd

Not Run / Remaining Gaps

• The three new cases are premerge: false. qwen35-9b stays the family's premerge gate so
  the premerge suite still builds one Qwen3.5 model rather than four; the new widths run on
  demand and through the release performance matrix.
• The committed E2E runs the reference on the GPU as every LLM family does. This dev container
  carries a CPU-only torch, so locally the reference ran on the CPU; the GPU form runs in CI.
• The three widths are listed in excluded_profiles, not benchmarked: the release-performance
  workload and receipt were collected only for the 9B, which exercises the same builder and
  runtime path.
• MoE Qwen3.5 checkpoints (35B-A3B, 122B-A10B, 397B-A17B) are out of scope here.

Contributor Self-Review

• I have completed a self-review of this change.

Notes For Future Readers

Worth recording that this is the first change in the repository to add a manifest to an
existing family without touching family code — every prior manifest arrived as part of the
commit that created its family. The reason it works here is that qwen3_5 was written to
claim a lineage rather than a checkpoint, and the widths differ only in layer and hidden
dimensions, which the builder already reads from config.

The routing check was the part worth doing carefully. Two families naming the same
model_type looks like a latent conflict, but the complementary output_gate_type /
mlp_only_layers predicates make it exact, and that is what decides which family owns a
future Qwen3.x release.

Third-party provenance: the checkpoints are the published Qwen/Qwen3.5-* Hugging Face
repositories.

Risk level

• Low

Test inputs only. No family code, no runtime code, and the premerge gate is unchanged.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f4de3d7c-0b3a-4b50-8eb8-5f49e947b2da

📥 Commits

Reviewing files that changed from the base of the PR and between df092fb and 79c38a1.

📒 Files selected for processing (1)

• apps/benchmark/performance/release.yaml

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

📝 Summary

Summary

Adds FP16 test manifests for Qwen3.5 0.8B, 2B, and 4B. Each manifest uses a Hugging Face checkpoint, chat templates, disabled thinking, a France-capital prompt, and single-device execution. All cases remain outside premerge.

Excludes these widths from release-performance evaluation. The release workload and receipt currently cover only Qwen3.5 9B.

Architecture impact

• Family-owned files: Three new manifests under families/qwen3_5/tests/manifests/.
• Shared surface: apps/benchmark/performance/release.yaml.
• Dependency directions: No new dependencies or family cross-dependencies.
• Affected consumers: On-demand manifest tests and the release-performance matrix.
• Unresolved blast-radius questions: Review findings and severity counts were not supplied. Test receipts were also not supplied.

Review outcome

HUMAN REVIEW REQUIRED

The objectives report 154 passing architecture and benchmark tests and token-for-token Hugging Face parity with NED 0.0000. The supplied shell output confirms the manifest and release configuration, but it does not verify those test results.

Walkthrough

Three Qwen3.5 text-generation manifests were added for the 0.8B, 2B, and 4B models. Release-performance exclusions were added for these widths because workload and receipt coverage exists only for the 9B width.

Changes

Qwen3.5 profiles

Layer / File(s)	Summary
Add Qwen3.5 model manifests families/qwen3_5/tests/manifests/qwen35-*.json	Added manifests with model metadata, FP16 settings, runtime limits, tensor-parallel configuration, and non-premerge chat-template tests.
Exclude smaller widths from release performance apps/benchmark/performance/release.yaml	Excluded the 0.8B, 2B, and 4B profiles from release-performance evaluation because only the 9B width has collected workload and receipt coverage.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 79c38

The new model profiles and performance exclusions match the repository’s catalog and release-suite contracts, with no identified merge-blocking risk.

🚥 Pre-merge checks | ✅ 8 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Shared Semantic Neutrality	⚠️ Warning	The pull request changes shared configuration in apps/benchmark/performance/release.yaml, outside the exempt family-owned test directory. It adds three model-specific excluded_profiles entries and…	Remove the Qwen-specific exclusions from the shared release suite, or supply the specialization through an existing narrow family-owned data contract that the shared loader consumes generically. Alternatively, add matching release-performan…

✅ Passed checks (8 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Family Ownership Boundary	✅ Passed	No cross-family ownership dependency is introduced. The three new validation artifacts are under families/qwen3_5/tests/manifests/ and each declares "family": "qwen3_5" (changed files, lines 1-22)…
Benchmark Validation Integrity	✅ Passed	No benchmark-validation integrity failure is introduced. The diff adds three manifests and excludes those models from release performance because no matching workload or receipt exists. The release su…
Shared Change Blast Radius	✅ Passed	The check applies because the PR changes the shared release-performance suite, but the required blast-radius evidence is present. The description identifies the generic need: the release matrix must n…
Title check	✅ Passed	The title clearly and concisely describes qualifying the remaining published Qwen3.5 widths, which is the primary change.
Description check	✅ Passed	The description covers the required background, exit criteria, implementation, change category, validation results, environment, remaining gaps, self-review, notes, and risk. It is complete enough for…

Full details: Shared Semantic Neutrality

Explanation

The pull request changes shared configuration in apps/benchmark/performance/release.yaml, outside the exempt family-owned test directory. It adds three model-specific excluded_profiles entries and a Qwen3.5-specific reason that records reference-parity evidence and release-workload/receipt status. tools/perf_matrix.py consumes these entries and subtracts the named models from ready-model coverage, so the change makes a shared release-matrix decision for qwen3_5 widths. The new manifests are family-owned, but they do not make the shared exclusion behavior model-agnostic.

Resolution

Remove the Qwen-specific exclusions from the shared release suite, or supply the specialization through an existing narrow family-owned data contract that the shared loader consumes generically. Alternatively, add matching release-performance workloads and receipts so no Qwen-specific exclusion is required.

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

This is an automated Internal CI result; no review from an individual maintainer is requested.

TRTMC Protected CI result
=========================

Status: FAILED
Pull request: #1289
Head commit: 79c38a1292d4e4db071281c9a4bfb66d2ac783b7
Reason: Automated internal CI failed; details withheld

Protected failure details are not transferred to the public repository.

Open the public Source Actions run from the automated status link above.