Skip to content

[AMD] dsv4-fp4-mi355x-sglang: switch fixed-seq-len search space to TP4 #1768

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
seungrokj merged 10 commits into from
Jun 16, 2026
+12 −3
Merged

[AMD] dsv4-fp4-mi355x-sglang: switch fixed-seq-len search space to TP4 #1768

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
c8541ac
[AMD] update dsv4-fp4-mi355x-sglang fixed-seq-len search space in mas…
seungrokj Jun 15, 2026
a1f0502
[AMD] update dsv4-fp4-mi355x-sglang fixed-seq-len search space: re-en…
seungrokj Jun 15, 2026
9628d11
[AMD] perf-changelog: dsv4-fp4-mi355x-sglang TP4 fixed-seq-len search…
seungrokj Jun 15, 2026
50b4a94
[AMD] narrow dsv4-fp4-mi355x-sglang fixed-seq-len to single isl=8192 …
seungrokj Jun 15, 2026
d7b8862
[AMD] add memory budget comments and set isl=1024 dp-attn conc=2048 p…
seungrokj Jun 15, 2026
4327aab
[AMD] clarify memory budget comments for dsv4-fp4-mi355x-sglang
seungrokj Jun 15, 2026
24bde2a
[AMD] expand dsv4-fp4-mi355x-sglang fixed-seq-len: add TP8/TP4 dp-att…
seungrokj Jun 15, 2026
c6daac8
Merge branch 'main' into amd/dsv4_sgl_0616
seungrokj Jun 15, 2026
64213b9
[AMD] remove stale memory budget comments from amd-master.yaml
seungrokj Jun 15, 2026
b519ea6
Merge branch 'main' into amd/dsv4_sgl_0616
Oseltamivir Jun 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions .github/configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2135,7 +2135,6 @@ dsr1-fp4-mi355x-sglang-disagg-8k1k-mtp:
- "DECODE_NODES=1"
- "DECODE_MTP_SIZE=1"


dsv4-fp4-mi355x-sglang:
image: lmsysorg/sglang-rocm:v0.5.13-rocm720-mi35x-20260612
model: deepseek-ai/DeepSeek-V4-Pro
Expand All @@ -2150,12 +2149,15 @@ dsv4-fp4-mi355x-sglang:
osl: 1024
search-space:
- { tp: 8, dp-attn: true, conc-start: 64, conc-end: 2048 }
- { tp: 8, dp-attn: false, conc-start: 1 , conc-end: 32 }
- { tp: 4, dp-attn: true, conc-start: 16, conc-end: 128 }

@cursor cursor Bot Jun 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TP4 dp-attn conc mismatch

Medium Severity

The new TP4 dp-attn: true fixed-seq-len rows use conc-start 16 and conc-end 128, but the same PR’s perf-changelog.yaml entry documents that sweep as concurrency 32–2048. The CI sweep will not cover the high-concurrency TP4 dp-attn range the changelog and PR describe.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit b519ea6. Configure here.

- { tp: 4, dp-attn: false, conc-start: 1 , conc-end: 32 }
- isl: 8192
osl: 1024
search-space:
- { tp: 8, dp-attn: true, conc-start: 64, conc-end: 2048 }
- { tp: 8, dp-attn: false, conc-start: 1, conc-end: 32 }
- { tp: 4, dp-attn: true, conc-start: 16, conc-end: 128 }
- { tp: 4, dp-attn: false, conc-start: 1, conc-end: 32 }


# MTP variant of dsv4-fp4-mi355x-sglang. Mirrors the base search space and adds
# spec-decoding: mtp, which routes to dsv4_fp4_mi355x_sglang_mtp.sh (EAGLE
Expand Down
7 changes: 7 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3831,6 +3831,13 @@
- "Extend MiniMax-M3 MXFP8 H100/H200 non-MTP sweeps to concurrency 1 on the latency rows (H100: TP8; H200: TP4 and TP8) and add full TEP coverage from conc 1 to 256 (H100: TP8+EP8; H200: TP4+EP4 and TP8+EP8, incl. a new TP4+EP4 row for 8k1k). H200 TP8+EP8 upper bound moves 512->256 (high concurrency stays covered by the TP8+EP8 dp-attn DEP rows). DEP rows unchanged"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1761

- config-keys:
- dsv4-fp4-mi355x-sglang
description:
- "Switch fixed-seq-len search space from TP8 to TP4 for both isl=1024 and isl=8192 scenarios"
- "Expand isl=8192 coverage: add TP4 dp-attn sweep (conc 32–2048) and TP4 TP-only sweep (conc 1–32)"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1762

- config-keys:
- dsv4-fp4-gb300-dynamo-trt
- dsv4-fp4-gb300-dynamo-trt-mtp
Expand Down