Validate archived MI355X MiniMax M3 vLLM disaggregation with srt-slurm

[cquil11]

Scope

Port the explicitly requested deprecated MiniMax-M3 FP4 MI355X vLLM disaggregated compatibility case. This remains archived and opt-in; it is not restored to active production sweeps. Depends on #2542 and srt-slurm PR #7 runtime at 33a3d811b7956973e4ef78b1d0cf4a5d56dad6a1. Only this compatibility lane selects the new runtime; previously validated recipes retain 81d46274. Latest InferenceX main c355f54 is included.

Preserved behavior

• Two exclusive MI355X nodes: one TP4 prefill worker and one TP4 decode worker, no EP/DP attention or speculation.
• All nine concurrency points: 1, 2, 4, 8, 16, 32, 64, 128, 256; original 8192/1024 lengths, 0.8 random range, warmups and request counts, and ten-second inter-point cooldown.
• All original explicit vLLM CLI options and model/role environment values were compared semantically against upstream models_vllm.yaml and server_vllm.sh. The same provisioned checkpoint is mounted at /model. No context-limit override is added.
• Native MoRI-IO read mode, native vLLM Router ZMQ discovery, and consistent-hash routing for both roles.
• Unchanged benchmark_serving.py through the custom benchmark interface. srt-slurm owns allocation, workers/router, health, logs, and teardown; no amd_utils, host-repair scripts, or independent Docker control plane.

Explicit image refresh

The original engine nightly-2dfaae752b4db0d43cfc0715c780e33be030d0f1 and router nightly-20260716-1fbcde7 tags both return 404 from Docker Hub. Use the official vLLM ROCm v0.28.0 release (digest sha256:e0a3b2bd3fe7ec563916c3a5d949898d133458c18d6b2f460c906885cfb32032) and official vLLM Router nightly-20260903-1d10e71 (digest sha256:be9325fc60f0a9cf896003ce40c7fb902cc834b1c0c0bdadc6259ba6759b5536).

This is an explicit dependency change requiring new performance/accuracy validation. No engine patches, cherry-picks, custom images, package replacement hooks, or setup scripts. The release already contains MoRI-IO and amd-quark.

Validation

Complete end-to-end run: 33837838363, attempt 1 — success. Matrix generation, throughput, eval, both collectors, and success-rate calculation all passed.

• Exact InferenceX checkout: 86b6762486f245c2ebdca170ee9efc23055f5358. The dispatch workflow came from main c355f549a; its separate checkout input selected this PR commit.
• Exact srt-slurm runtime: 33a3d811b7956973e4ef78b1d0cf4a5d56dad6a1.
• Throughput: all nine points, concurrency 1 through 256, 5,116/5,116 measured requests. Raw audit verified distinct point coverage, exact request/token totals, nonempty outputs, correct model identity, and finite metrics.
• Full GSM8K at concurrency 256: 1,319 unique questions, no evaluation limit, zero empty responses. Strict accuracy 0.9469294920394238, flexible accuracy 0.9461713419257013; unchanged threshold 0.90. Per-filter sample averages exactly match the native result.
• Collector audit: all nine throughput rows preserve TP4/TP4, one worker per role, four prefill plus four decode GPUs, original workload dimensions, official image, native router identity, and MoRI transfer metadata. Per-GPU output throughput matches the raw measurements. Eval collector reports n_eff=1319, infrastructure_success=true, and no integration error.
• Slurm eval 42262 and throughput 42263 both ended COMPLETED with exit 0:0. No host-repair steps or manual process cleanup were needed.
• Additional repeatability evidence: earlier throughput job independently passed the same nine points and 5,116 requests on the same source/runtime. That workflow's eval had an invalid dispatch input, so it is not counted as a green full run or used to fill gaps in the successful run above.
• Power is not validated (power_valid=0). The optional srt dashboard does not contain custom-client points; canonical benchmark/eval artifacts, not that dashboard, are the validation source.
• The official engine emits a READ-mode/full-graph warning. The preserved configuration passed the actual full accuracy test; no graph-mode override was introduced.

Resolved issues

1. The initial runtime run failed during MoRI initialization because srt-slurm injected one VLLM_PORT scan base inherited by all TP children. srt-slurm c0472a82, incorporated into PR [NVIDIA] Add DSR1 TensorRT Support and Enhanced Plotting #7 at 33a3d811, preserves native ephemeral-port allocation for MoRI. Non-MoRI rendezvous behavior is unchanged. This is an orchestration fix, not an engine patch.
2. The next dispatch incorrectly passed literal eval-limit=full. The lm-eval path requires the empty default for the full split. Correcting the dispatch input required no source, engine, or workload change. The successful run above completed independently with the empty limit.

Local checks and regression boundary

• Strict matrix generation and recipe dry-run passed. The archived lane requests exactly two physical nodes; its nine points and full eval were verified.
• 360 existing InferenceX matrix/changelog tests passed.
• Combined runtime: 112 focused ATOM/vLLM/router/accelerator/dry-run tests passed. Two MoRI regression cases failed before the source fix and passed after it.
• Broad local runtime suite: 1546 passed, 2 skipped, 6 deselected; five failures also reproduced on the untouched 81d46274 baseline (three sa-bench shell-path tests, one local shell-profile test, one Linux-only affinity mock on macOS). The full local suite is not claimed green.
• No previously validated recipe or runtime pin was changed. This archived lane alone selects runtime 33a3d811; existing validated recipes retain 81d46274.

Review boundary

This remains a draft archived compatibility port, not a production reactivation. The normal performance-changelog processor only resolves active master-config keys and rejects this archived key. This PR does not bypass that guard, remove its required changelog, or restore deprecated production sweeps.

The successful validation is a manual archived-config e2e-tests.yml run, not a native PR run-sweep.yml run. It is not eligible for /stage-results or sweep reuse, and the PR is not claimed merge-ready.

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[cquil11]

Runtime startup now passes with the clean MoRI port-allocation fix: both allocations completed all TP handshakes, routed generation, and backend health checks on the unchanged official images.

Run 33837126410 exposed a dispatch-input mistake before evaluation began: literal eval-limit=full is passed through to lm-eval as --limit full, which is invalid. The full dataset requires the empty default for this harness. No engine, benchmark, or srt-slurm source change is needed for that error.

Corrected full throughput-plus-eval run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33837838363
Exact checkout: 86b6762. Runtime: 33a3d811b7956973e4ef78b1d0cf4a5d56dad6a1. All nine concurrency points; eval-limit empty. The prior throughput job is left intact as additional evidence, not combined with this run to claim full-run success. Validation remains in progress.

[cquil11]

The first full throughput sweep has passed: all nine concurrency points and 5,116/5,116 measured requests completed on the official vLLM v0.28.0 image. Throughput job: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33837126410/job/100911912515

The raw artifact audit passed for unique point coverage, exact request/token totals, nonempty outputs, and finite metrics. srt-slurm completed and released the allocation without any host-repair steps.

This is not yet complete validation: that workflow still fails because of the earlier eval-limit dispatch mistake. The corrected full run remains https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33837838363, with full GSM8K (empty limit) and all nine throughput points. Its eval is prioritized first in the MI355X queue. No engine patches, config workarounds, or changes to previously validated recipes have been introduced.

[cquil11]

Full accuracy validation is now green on the corrected run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33837838363/job/100913948095

• Full GSM8K: 1,319 unique documents in each scoring filter; no evaluation limit.
• Strict accuracy: 0.9469294920394238. Flexible accuracy: 0.9461713419257013. Both exceed the unchanged 0.90 threshold.
• Per-sample audit: zero empty responses, unique document IDs/hashes, and recomputed averages exactly matching the native result.
• The collected artifact agrees: n_eff=1319, infrastructure_success=true, integration_error=null, concurrency 256, TP4/TP4.

No graph-mode override, engine patch, or config workaround was needed. The same corrected workflow is now running its full nine-point throughput job (Slurm 42263), so whole-run validation is still in progress. The earlier nine-point throughput sweep also passed and was independently audited, but is not being used as a fill-in for this run.

[cquil11]

Complete validation is green: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33837838363 (attempt 1).

All nine throughput points passed with 5,116/5,116 measured requests. Full GSM8K passed with 1,319 unique questions, zero empty responses, strict accuracy 0.9469294920394238, and flexible accuracy 0.9461713419257013 against the unchanged 0.90 threshold. Raw and collected artifacts were independently checked for counts, token totals, finite metrics, topology/GPU metadata, and matching scores. Both Slurm allocations completed 0:0 and released normally.

Exact tested checkout: 86b6762; srt-slurm: 33a3d811b7956973e4ef78b1d0cf4a5d56dad6a1. Official images only, no engine patches, graph-mode override, configuration workaround, or fill-in run.

The PR description now contains the complete evidence and caveats. It remains draft solely as an archived compatibility port: the regular changelog processor does not accept archived keys, and this manual e2e run is not stageable or reusable as a native PR sweep. No deprecated production sweep was re-enabled.