Skip to content

[AMD][DSV4] Repin MI355X vLLM entries to the nightly shipping FHMoE #2850

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Fangzhou-Ai wants to merge 1 commit into from
Closed

[AMD][DSV4] Repin MI355X vLLM entries to the nightly shipping FHMoE #2850

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1202,7 +1202,7 @@ dsv4-fp4-mi355x-sglang-mtp:
# gpu-mem-util=0.6. TP8 sweeps conc 4-64; DEP8 has a single conc=64
# probe to validate the ROCm DP+EP path.
dsv4-fp4-mi355x-vllm:
image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa
image: vllm/vllm-openai-rocm:nightly-de69e821b7c834bcf1ec96325ec8c410ac183680
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: mi355x
Expand All @@ -1225,7 +1225,7 @@ dsv4-fp4-mi355x-vllm:
# above ~conc32 (-37% @ conc32). Image reuses the base entry's v0.22.0 ROCm
# build, which already contains the MTP commit.
dsv4-fp4-mi355x-vllm-mtp:
image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa
image: vllm/vllm-openai-rocm:nightly-de69e821b7c834bcf1ec96325ec8c410ac183680
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: mi355x
Expand Down Expand Up @@ -1316,7 +1316,7 @@ qwen3.5-fp8-mi325x-sglang-mtp:
- { tp: 8, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp }

dsv4-fp4-mi355x-vllm-agentic-mtp:
image: vllm/vllm-openai-rocm:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0
image: vllm/vllm-openai-rocm:nightly-de69e821b7c834bcf1ec96325ec8c410ac183680
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: cluster:mi355x-amds
Expand Down
13 changes: 13 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6902,3 +6902,16 @@
- "Lower mem-fraction-static from 0.89 to 0.86 on every arm: TP4, TP8 and TP8 with DP attention. swa-full-tokens-ratio becomes per-arm: 0.10 on the tensor-parallel arms as before, 0.15 under DP attention."
- "Drop concurrency 2 and 10 from the TP4 arm, leaving [1, 4, 8], and drop concurrency 16 from the TP8 hicache arm, leaving [32, 48]. Concurrency 16 remains on the TP8 no-offload arm."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2800

- config-keys:
- dsv4-fp4-mi355x-vllm
- dsv4-fp4-mi355x-vllm-mtp
- dsv4-fp4-mi355x-vllm-agentic-mtp
scenario-type:
- fixed-seq-len
- agentic-coding
description:
- "Repin all three DSv4 MI355X vLLM entries to the first ROCm nightly that ships vllm-project/vllm#53161 (vllm-project/vllm@de69e821), which fuses the DeepSeek-V4 native-FP8 shared expert into the MXFP4 routed-expert AITER kernel (FHMoE)."
- "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 was already exported by the AgentX script and is added to the 8K/1K STP and MTP scripts by #2792, but on every nightly before de69e821 the request was inert: vLLM's eligibility check rejected this mixed FP4+FP8 checkpoint and the fusion self-disabled at startup. #53161 is what makes the flag take effect."
- "Fusion eligibility is gated on gfx950, tensor parallelism 8, data parallelism 1, no expert parallelism and no EPLB, --moe-backend aiter, BF16 model dtype, and the DeepSeek-V4 block-128 FP8 shared-expert quantization config. All three entries satisfy it; the AgentX DP-attention arm does not and keeps the unfused path."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PENDING