[https://nvbugs/6691772][fix] Remove the TLLM_K3_MLA_GEN_BACKEND=trtllm-gen pin from worker_env_var so…

[trtllm-agent]

Summary

• Root cause: The perf config pins TLLM_K3_MLA_GEN_BACKEND=trtllm-gen while enable_attention_dp: true replicates all 96 Q heads on every rank, and trtllm-gen MLA decode rejects 64 < num_heads_q < 128.
• Fix: Remove the TLLM_K3_MLA_GEN_BACKEND=trtllm-gen pin from worker_env_var so the module default cute-dsl (the tuned H=96 BF16-KV backend) applies, and update the comment block that named the knob.
• Original test: python .repair-bot/repro.py
• Automated fix generated by repair-bot

Test plan

• Verify fix on the same GPU type as the original failure
• Check for regressions in related tests

Links

• Bug: https://nvbugs/6691772

Reproduction comparison

• Failed commit: pending
• ToT: repro_on_tot at f221314
  Signature: ValueError: Kimi K3 MLA: the trtllm-gen generation backend cannot run with 96 query heads per rank (trtllm-gen MLA decode rejects 64 < num_heads_q < 128; under attention-DP every rank keeps all heads). trtllm-gen was selected explicitly — by the FP8-KV-cache override or by TLLM_K3_MLA_GEN_BACKEND=trtllm-gen. Use tensor-parallel head sharding (TEP) so each rank has <= 64 heads, or a BF16 KV cache with the default cute-dsl backend.
• Signature relation: pending

Dev Engineer Review

• Removed the unsupported TLLM_K3_MLA_GEN_BACKEND=trtllm-gen override from the Kimi K3 DEP16 disaggregated performance configuration.
• Updated the related comment to document that attention-DP replicates all 96 query heads and uses the module-default cute-dsl backend.
• Other worker environment variables remain unchanged.
• The configuration change is scoped to the affected BF16 KV-cache workload.

QA Engineer Review

No test changes.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7b0ad28c-39cb-49b2-a768-a170a701337e

📥 Commits

Reviewing files that changed from the base of the PR and between 0699c4b and bbd8276.

📒 Files selected for processing (1)

• tests/scripts/perf/disaggregated/gb300_kimi-k3-fp4_8k1k_con512_ctx1_dep16_gen1_dep16_eplb0_mtp0_ccb-NIXL.yaml

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The worker environment configuration removes the explicit trtllm-gen MLA backend setting. The runtime now uses the module-default backend. Other context and generation worker variables remain unchanged.

Changes

Worker backend configuration

Layer / File(s)	Summary
Use the module-default MLA backend tests/scripts/perf/disaggregated/gb300_kimi-k3-fp4_8k1k_con512_ctx1_dep16_gen1_dep16_eplb0_mtp0_ccb-NIXL.yaml	Removes the explicit TLLM_K3_MLA_GEN_BACKEND=trtllm-gen setting while preserving the remaining worker environment variables.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Merge Risk: ⚪ Minimal · up to bbd82

This PR removes an explicit backend pin from a benchmark configuration so the module default can be used. No concrete correctness, deployment, or availability risk remains at the current head; it is merge-ready after normal checks.

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the NVBugs fix and the primary change: removing the TLLM_K3_MLA_GEN_BACKEND=trtllm-gen pin. The trailing ellipsis is slightly awkward but does not obscure the change.
Description check	✅ Passed	The description explains the root cause, fix, affected configuration, reproduction details, test plan, and bug link. It uses Summary and Test plan headings instead of the template's Description and Te…
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description explains the root cause, fix, affected configuration, reproduction details, test plan, and bug link. It uses Summary and Test plan headings instead of the template's Description and Test Coverage headings, but the required information is present.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[trtllm-agent]

[Repair Bot][Two-Leg Repro Comparison:6691772-f221314f60af-1788316854606643536]

Reproduction comparison:

• Failed commit: no_repro_on_failed_commit at 785c948
• ToT: repro_on_tot at f221314
  Signature: ValueError: Kimi K3 MLA: the trtllm-gen generation backend cannot run with 96 query heads per rank (trtllm-gen MLA decode rejects 64 < num_heads_q < 128; under attention-DP every rank keeps all heads). trtllm-gen was selected explicitly — by the FP8-KV-cache override or by TLLM_K3_MLA_GEN_BACKEND=trtllm-gen. Use tensor-parallel head sharding (TEP) so each rank has <= 64 heads, or a BF16 KV cache with the default cute-dsl backend.
• Signature relation: only_tot

[sylvesterkaczmarek]

I checked this against the current Kimi K3 backend-selection contract. This recipe does not request FP8 KV cache (kv_cache_config.dtype remains auto), and the K3 documentation still lists FP8 KV cache as unsupported; for the supported BF16 KV path, K3's generation MLA default is cute-dsl unless TLLM_K3_MLA_GEN_BACKEND explicitly overrides it.

With enable_attention_dp: true, all 96 K3 query heads remain present on each rank, so removing the explicit trtllm-gen pin lets the model select the backend that supports that head count without changing the DEP16 topology or the rest of the worker runtime environment.

I don't see a blocking issue in this config change.