[AMD][DSV4] Disagg AgentX DP-attention tuning, HiCache ratio investigation, FP4 sparse-attention indexer - #2823
[AMD][DSV4] Disagg AgentX DP-attention tuning, HiCache ratio investigation, FP4 sparse-attention indexer#2823ichbinblau wants to merge 12 commits into
Conversation
…perts fusion and chunked-prefill sizing - models.yaml: remove the leftover --disable-shared-experts-fusion that was silently overriding --enforce-shared-experts-fusion in base_flags (the loader gives disable unconditional priority), and make prefill.no_dp.chunked_prefill_size TP-conditional (16384 @ TP8, 8192 @ TP4) to fix the conc>=16 queue-saturation/decode-stall failure mode, mirroring the single-node dsv4-fp4-mi355x-sglang-agentic-mtp fix (#2784). - env.sh: comment out DSv4 kernel-routing env vars that now match v0.5.18 defaults, add TORCH_BLAS_PREFER_HIPBLASLT and SGLANG_OPT_USE_AITER_BATCHED_GEMM, lower GPU_MAX_HW_QUEUES from 5 to 2. - amd-master.yaml: bump dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp to image v0.5.18-rocm720-mi35x-20260829 and add a TP4/EP1 no-offload arm at conc-list [1,2,4,8,16]. - perf-changelog.yaml: record the above. Co-authored-by: Cursor <cursoragent@cursor.com>
… bump image - env.sh: only export the DP-attention-only SGLang knobs (SGLANG_SHARED_EXPERT_TP1, SGLANG_DP_SHARED_EXPERT_LOCAL, SGLANG_DP_USE_GATHERV, SGLANG_DP_USE_REDUCE_SCATTER, wider GPU_MAX_HW_QUEUES) when PREFILL_ENABLE_DP or DECODE_ENABLE_DP is true, so TP-only arms keep GPU_MAX_HW_QUEUES=2 instead of inheriting the DP tuning unconditionally. Also add SGLANG_MORI_RECV_BOUND=1 for the disagg MoRI path. - models.yaml: round out dp_flags with --tokenizer-worker-num 8, --enable-dp-attention-local-control-broadcast, --stream-interval 20, and --prefill-decode-interval 10 to mirror the single-node recipe's DP-attention block; add enable_two_batch_overlap under prefill.dp; raise mem_fraction_static 0.85 -> 0.90 for both prefill and decode. - amd-master.yaml: bump the image to v0.5.18-rocm720-mi35x-20260831, switch the active search-space arm's decode side from ep:8 to ep:1 (DP-attention over TP-sharded MoE instead of EP), and widen its conc-list to [64, 96, 128, 192, 256]. Keep the TP4/TP8 no-DP arms as commented-out reference. Co-authored-by: Cursor <cursoragent@cursor.com>
…ags/no_dp_flags buckets sglang documents --enforce-shared-experts-fusion and --disable-shared-experts-fusion as mutually exclusive, but DeepSeek-V4-Pro-AgentX's base_flags carried the enforce flag unconditionally while dp_flags added the disable flag on top -- both landed on the same command line for the DP arm, only resolving correctly because DeepseekV4ForCausalLM.determine_num_fused_shared_experts() happens to check disable_shared_experts_fusion first. That's an internal implementation detail, not a guaranteed contract, and reads as a contradiction to anyone inspecting the launch command. - models.yaml: add a new no_dp_flags field (mirrors dp_flags, applied when DP attention is off) and move --enforce-shared-experts-fusion out of base_flags into it, so the two conflicting flags are never emitted together for any arm. - server_sglang.sh: parse MODEL_NO_DP_FLAGS and apply it in build_server_config() whenever enable_dp is false, alongside the existing dp_config branch. Co-authored-by: Cursor <cursoragent@cursor.com>
DeepSeek-V4-Pro-AgentX's decode.dp arm now serves concurrency up to 256 (configs/amd-master.yaml conc-list), so capture CUDA graphs up to bs=256 instead of 128 to keep the graph replay path covering the full sweep. Co-authored-by: Cursor <cursoragent@cursor.com>
Scope the sweep to just the highest concurrency point while iterating on the DP-attention arm, instead of re-running the full [64, 96, 128, 192, 256] sweep each time. Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
ratio=6 (eef805c) eliminated host-pool saturation but regressed throughput -11% and TTFT +65%/+107% (mean/p90) vs ratio=3 at conc=256: avg host-pool occupancy barely moved (~15.5M tokens either way), so the extra capacity was mostly unused while lookup/restore cost scaled with the larger pool, hurting prefill-side TTFT specifically. ratio=4 targets the ~25M-token peak observed on the busiest rank (above ratio=3's ~19.5M cap) without doubling pool size like ratio=6. Co-authored-by: Cursor <cursoragent@cursor.com>
- DP-attention gating, EP1 tuning, and image bump to ...20260831 - Mutually-exclusive no_dp_flags/dp_flags split for shared-experts fusion - Widened decode cuda-graph bs range and conc-list narrowing - HICACHE_RATIO 3->6->4 investigation and rationale - FP4 sparse-attention indexer + image bump to ...20260903 (sglang PR #37353) Co-authored-by: Cursor <cursoragent@cursor.com>
…full search space - models.yaml: add --enable-deepseek-v4-fp4-indexer to DeepSeek-V4-Pro-AgentX base_flags now that sglang gfx95 support has landed upstream (PR #37353); lower prefill/decode mem_fraction_static 0.90 -> 0.86 to make room for the larger FP4 indexer working buffers within the same GPU memory budget. - amd-master.yaml: bump the disagg AgentX HiCache image from v0.5.18-rocm720-mi35x-20260831 to ...20260903 (first tag carrying the FP4 indexer gfx95 kernels). Restore the previously commented-out TP4/TP8 no-DP offload arms (conc-list [1,4,8,10] and [16]) and the TP8 HiCache no-DP arm (conc-list [32,48,64]) as active search-space entries, and widen the DP-attention arm's conc-list back from [256] to the full [64, 96, 128, 192, 256] sweep now that FP4-indexer-driven KV headroom changes the capacity picture. Drop the HICACHE_RATIO=4 override (and its explanatory comment) so the arm falls back to server_sglang.sh's default ratio, since the FP4 indexer's smaller per-token KV footprint (68B vs 132B) changes the sizing math behind that ratio=3->6->4 investigation. Co-authored-by: Cursor <cursoragent@cursor.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| export SGLANG_DP_USE_GATHERV=1 | ||
| export SGLANG_DP_USE_REDUCE_SCATTER=1 | ||
| export GPU_MAX_HW_QUEUES=5 | ||
| # export SGLANG_EAGER_INPUT_NO_COPY=true |
There was a problem hiding this comment.
Shared DSv4 crash knobs commented out
Medium Severity
The shared DeepSeek-V4-Pro env block comments out the kernel-routing exports that the surrounding comment still describes as required to avoid a fused-MoE heuristic-dispatch crash at decode. That block is selected by MODEL_NAME, so DeepSeek-V4-Pro-DI loses the same pins even though this PR only retunes AgentX.
Reviewed by Cursor Bugbot for commit 07ce2fb. Configure here.
ichbinblau
added
AMD
agentx
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Beyond the inline finding, I also checked the DP arm's new chunked_prefill_size formula for TP8 (16384 * PREFILL_TP_SIZE, i.e. 131072) against the "compressor kernel uint16 token cap (255*256)" comment retained on that line — the actual per-rank dispatch cap enforced by the mori kernel is governed separately by MORI_MAX_DISPATCH_TOKENS_PREFILL in env.sh (unchanged at 8192), so chunked_prefill_size doesn't feed that limit and the doubling isn't a functional regression, just a stale comment.
Extended reasoning...
This is the brief findings-present ruled-out note: one inline finding (perf-changelog.yaml missing a mem_fraction_static entry) is already reported. I additionally traced the DP arm's chunked_prefill_size formula change (models.yaml prefill.dp, ~line 392) that doubles to 131072 for TP8 versus the old MORI_MAX_DISPATCH_TOKENS_PREFILL-derived value, and confirmed via server_sglang.sh/env.sh that the actual per-rank mori dispatch cap referenced by the adjacent comment is controlled by the separate, unchanged env var MORI_MAX_DISPATCH_TOKENS_PREFILL=8192, not by chunked_prefill_size itself — so the doubling does not violate the documented compressor-kernel constraint, it's just a stale comment left over from when the two were tied together.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ff29006. Configure here.
| # --swa-full-tokens-ratio 0.15 widens the SWA ring for the larger DP batch, and | ||
| # --tokenizer-worker-num matches the only DP arm's TP (8) today -- bump this if | ||
| # a different-TP DP arm is added. | ||
| dp_flags: "--enable-dp-attention --enable-prefill-delayer --disable-shared-experts-fusion --swa-full-tokens-ratio 0.15 --enable-dp-attention-local-control-broadcast --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10" |
There was a problem hiding this comment.
Colocated PD interval on disagg
Medium Severity
dp_flags now includes --prefill-decode-interval 10 for both disagg roles. That flag defers the next prefill for N scheduler rounds so colocated decode can run. On a prefill-only PD server every batch is prefill, so this inserts idle rounds after each chunk and can stall prefill throughput.
Reviewed by Cursor Bugbot for commit ff29006. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33837253408 |
3 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33837253408 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33837253408 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33837253408 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34074758948 |
1 participant
Summary
Tunes the
dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp(DeepSeek-V4-Pro-AgentX, MI355X, disaggregated, HiCache DRAM offload) recipe:DP-attention rework
SGLANG_SHARED_EXPERT_TP1,SGLANG_DP_SHARED_EXPERT_LOCAL,SGLANG_DP_USE_GATHERV,SGLANG_DP_USE_REDUCE_SCATTER, widerGPU_MAX_HW_QUEUES) behindPREFILL_ENABLE_DP/DECODE_ENABLE_DPinenv.sh, instead of exporting them unconditionally, so TP-only arms keepGPU_MAX_HW_QUEUES=2. AddSGLANG_MORI_RECV_BOUND=1for the disagg MoRI path.dp_flagsinmodels.yaml(--tokenizer-worker-num 8,--enable-dp-attention-local-control-broadcast,--stream-interval 20,--prefill-decode-interval 10,enable_two_batch_overlap) to mirror the single-node DP-attention recipe.ep:8toep:1(DP-attention over TP-sharded MoE instead of expert-parallel dispatch).--enforce-shared-experts-fusion/--disable-shared-experts-fusioninto mutually exclusiveno_dp_flags/dp_flagsbuckets (newno_dp_flagsfield inmodels.yaml,MODEL_NO_DP_FLAGSinserver_sglang.sh'sbuild_server_config()). Both flags previously rode the same command line for the DP arm and only resolved correctly by relying on an internal, undocumented check-order inDeepseekV4ForCausalLM.determine_num_fused_shared_experts().decode.dp.cuda_graph_bs_rangefrom1-128to1-256and the DP-attention arm'sconc-listback to the full[64, 96, 128, 192, 256]sweep.FP4 sparse-attention indexer
lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260831to...20260903, the first tag carrying gfx95 (MI350X/MI355X) support for DeepSeek-V4's FP4 sparse-attention indexer via AITER kernels (sglang PR #37353, merged 2026-09-02). Previously the flag was gated to NVIDIA SM100/SM120 only and hard-failed on AMD.--enable-deepseek-v4-fp4-indexertoDeepSeek-V4-Pro-AgentXbase_flags.mem_fraction_static0.90 -> 0.86 to make room for the FP4 indexer's working buffers within the same GPU memory budget.Note
Medium Risk
Changes disaggregated serving flags, MoE/DP env gating, and benchmark topology (EP→TP MoE under DP-attention); misconfiguration could affect correctness or stability at high concurrency, but scope is benchmark harness and recipe tuning rather than core app logic.
Overview
Retunes the dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp multinode recipe: bumps the SGLang ROCm image to v0.5.18 …20260903, enables
--enable-deepseek-v4-fp4-indexer, and adjusts mem_fraction_static to 0.86 on prefill/decode for AgentX.Harness / launch behavior: Adds a
no_dp_flagsbucket inmodels.yamlandMODEL_NO_DP_FLAGSinserver_sglang.shso--enforce-shared-experts-fusion(TP-only) and--disable-shared-experts-fusion(DP) never land on the same command line. Expandsdp_flagsto mirror the single-node DP-attention block (tokenizer workers, stream/prefill-decode intervals, wider SWA ratio, two-batch overlap on prefill DP).env.shgates DP-only collectives andGPU_MAX_HW_QUEUESonPREFILL_ENABLE_DP/DECODE_ENABLE_DP, setsSGLANG_MORI_RECV_BOUND=1, trims commented kernel overrides, and turns on AITER batched GEMM / hipBLASLt prefs.Search space (
amd-master.yaml): Splits concurrency into TP4 / TP8 no-DP and HiCache arms; DP-attention arm usesep: 1instead ofep: 8and sweepsconc-listup to 256; widens decode DP CUDA graph range to 1–256 and TP-conditional chunked_prefill_size (16384 @ TP8).Reviewed by Cursor Bugbot for commit 49c7646. Bugbot is set up for automated code reviews on this repo. Configure here.