Skip to content

Use DSpark6 for B200 DSV4 SGLang AgentX / B200 DSV4 SGLang AgentX 使用 DSpark6 #2821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nvpohanh wants to merge 2 commits into
base: main
Choose a base branch
from
+26 −12
Open

Use DSpark6 for B200 DSV4 SGLang AgentX / B200 DSV4 SGLang AgentX 使用 DSpark6 #2821

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
abc3480
feat(sglang): use DSpark for DSV4 B200 AgentX
nvpohanh Sep 4, 2026
874e8a6
chore: link perf changelog to #2821
nvpohanh Sep 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions benchmarks/single_node/agentic/dsv4_fp4_b200_sglang_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
set -eo pipefail
set -x

# Agentic trace replay for DeepSeek-V4-Pro FP4 on B200 with native EAGLE MTP.
# Agentic trace replay for DeepSeek-V4-Pro-0813 FP4 on B200 with DSpark K=6.
# Throughput uses the committed golden synthetic AL; eval retains real target
# verification.
#
Expand Down Expand Up @@ -110,8 +110,6 @@ CHUNKED_PREFILL_SIZE=8192
SWA_FULL_TOKENS_RATIO=0.1
MEM_FRACTION_STATIC=0.90
if [ "$DP_ATTENTION" = "true" ]; then
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320

# Leave HBM headroom for the FP4 indexer's context-dependent workspace.
Expand All @@ -132,12 +130,14 @@ if [ "$DP_ATTENTION" = "true" ]; then
--tokenizer-worker-num "$TP"
--prefill-decode-interval "$PREFILL_DECODE_INTERVAL"
--enable-dp-attention
--enable-dp-lm-head
--enable-dp-attention-local-control-broadcast
--incremental-streaming-output
--stream-interval 20
--dist-init-addr "127.0.0.1:$((PORT + 2000))"
--ep-size "$EP_SIZE"
--moe-a2a-backend megamoe
--enable-w4a4-mxfp4-megamoe
--enable-deepseek-v4-fp4-indexer
--disable-shared-experts-fusion
--disable-flashinfer-autotune
Expand Down Expand Up @@ -188,7 +188,7 @@ export SGLANG_OPT_USE_JIT_INDEXER_METADATA=1
export SGLANG_OPT_USE_TOPK_V2=1
export SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2=1
if [ "${EVAL_ONLY}" != "true" ]; then
export SGLANG_SIMULATE_ACC_LEN=2.49
export SGLANG_SIMULATE_ACC_LEN=3.77
export SGLANG_SIMULATE_ACC_METHOD=match-expected
export SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token
fi
Expand Down Expand Up @@ -218,10 +218,11 @@ SGLANG_CMD=(
--reasoning-parser deepseek-v4
--chat-template "$SCRIPT_DIR/../chat_templates/deepseek_v4_thinking.jinja"
--watchdog-timeout 1800
--speculative-algorithm EAGLE
--speculative-num-steps 3
--speculative-algorithm DSPARK
--speculative-dspark-block-size 6
--speculative-num-steps 1
--speculative-eagle-topk 1
--speculative-num-draft-tokens 4
--speculative-num-draft-tokens 7
# The B200 checkpoint lives on Lustre. Partition sequential prefetching
# across local ranks so post-load weight repacking reads from page cache
# instead of issuing redundant fragmented mmap faults from every rank.
Expand Down
8 changes: 4 additions & 4 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -930,7 +930,7 @@ dsv4-fp4-b200-sglang:

dsv4-fp4-b200-sglang-agentic-hicache-mtp:
image: lmsysorg/sglang:nightly-dev-20260827-20621aa1
model: deepseek-ai/DeepSeek-V4-Pro
model: deepseek-ai/DeepSeek-V4-Pro-0813
model-prefix: dsv4
runner: cluster:b200-nscale
precision: fp4
Expand All @@ -940,9 +940,9 @@ dsv4-fp4-b200-sglang-agentic-hicache-mtp:
agentic-coding:
- dram-utilization: 0.80
search-space:
- { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 3, 4, 5] }
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [8, 10, 16] }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [64, 96, 128, 160], router: { name: sglang-router, version: "0.3.2" } }
- { tp: 8, kv-offloading: none, spec-decoding: draft_model, conc-list: [1, 2, 3, 4, 5] }
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: draft_model, conc-list: [8, 10, 16] }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: draft_model, conc-list: [64, 96, 128, 160], router: { name: sglang-router, version: "0.3.2" } }
Comment on lines +943 to +945

@claude claude Bot Sep 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) These three search-space arms now set spec-decoding: draft_model, the first real usage of that value in nvidia-master.yaml, but configs/ci-priority.yaml's spec-decoding adjustments only list mtp/eagle/eagle3 (utils/ci_priority.py:163 does adjustments['spec-decoding'].get(value, 0)). CI priority scoring for this recipe's jobs silently drops by 0.75 (previously granted under spec-decoding: mtp) versus base branch, deprioritizing them in the run-sweep/e2e-tests/profile queues with no error. Fix: add draft_model to configs/ci-priority.yaml's spec-decoding adjustments (or otherwise ensure every spec-decoding value declared in nvidia-master.yaml has a matching CI-priority entry).

Extended reasoning...

utils/ci_priority.py calculate_priority() reads entry.get('spec-decoding','') and looks it up in policy['adjustments']['spec-decoding'] (configs/ci-priority.yaml lines 19-22: only mtp, eagle, eagle3 defined) via a plain dict .get(..., 0) fallback -- no validation/error for unknown values. Before this diff, these three search-space entries (tp:8 kv-offloading:none; tp:8 hicache; tp:8 ep:8 dp-attn hicache) declared spec-decoding: mtp and received +0.75. After the rename to draft_model, the same lookup returns 0, silently subtracting 0.75 from a total score of roughly 3.5-4.25 (base 1.0 + agentic 1.0 + precision fp4 0.75 + model-prefix dsv4 0.75 [+ multi-node 1.25 if applicable] + spec-decoding). This is invoked from .github/workflows/run-sweep.yml, e2e-tests.yml, profile.yml and speedbench-al.yml, so every CI run that schedules this recipe's jobs now ranks them lower relative to siblings still labeled mtp, with no warning that the policy file was not updated alongside the recipe rename.

Verification: nit. The mechanism is real and reachable, but its only consequence is CI queue-ordering, not a functional break. configs/nvidia-master.yaml:943-945 changes three agentic-coding search-space arms from spec-decoding: mtp to spec-decoding: draft_model. configs/ci-priority.yaml:19-22 defines spec-decoding adjustments only for mtp: 0.75, eagle: 0.75, eagle3: 0.75 — draft_model is not…


dsv4-fp4-b200-vllm:
image: vllm/vllm-openai:nightly-7c5dc571cbd1064ecc8a9b1045637ff647aa22cb
Expand Down
11 changes: 11 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6868,3 +6868,14 @@
description:
- "Refresh the DeepSeek-V4-Pro FP4 B200 vLLM image and search space, pin the Nscale NVFP4 checkpoint, use the FlashInfer CuTeDSL MoE backend for expert-parallel runs with EPLB disabled, set GPU memory utilization to 0.95, and cap the model length at 12288."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2534

- config-keys:
- dsv4-fp4-b200-sglang-agentic-hicache-mtp
scenario-type:
- agentic-coding
description:
- "Replace the B200 DeepSeek-V4-Pro AgentX native EAGLE/MTP serving path with DSpark block size 6 on the DeepSeek-V4-Pro-0813 checkpoint."
- "Use one speculative step, seven draft tokens, and thinking-on golden synthetic acceptance length 3.77 for throughput; eval retains real DSpark verification."
- "Use SGLang's current W4A4 MegaMoE and DP LM-head flags for DP-attention instead of the deprecated MegaMoE environment variables."
- "Resolve draft-model jobs through the existing B200 SGLang speculative recipe and the staged DeepSeek-V4-Pro-0813 checkpoint."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2821
4 changes: 3 additions & 1 deletion runners/launch_b200-nscale-compat.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ if [[ $MODEL_PREFIX == "dsr1" && $PRECISION == "fp4" ]]; then
elif [[ $MODEL_PREFIX == "dsr1" && $PRECISION == "fp8" ]]; then
export MODEL_PATH="/scratch/models/DeepSeek-R1-0528"
export SRT_SLURM_MODEL_PREFIX="dsr1-fp8"
elif [[ $MODEL_PREFIX == "dsv4" && $PRECISION == "fp4" && $MODEL == "deepseek-ai/DeepSeek-V4-Pro-0813" ]]; then
export MODEL_PATH="${MODEL_PATH:-/scratch/models/DeepSeek-V4-Pro-0813}"
elif [[ $MODEL_PREFIX == "dsv4" && $PRECISION == "fp4" ]]; then
# Node-local weights are not visible on the runner/login node.
export MODEL_PATH="/scratch/models/DeepSeek-V4-Pro-NVFP4"
Expand Down Expand Up @@ -540,7 +542,7 @@ else

SQUASH_FILE="/data/home/sa-shared/containers/$(echo "$IMAGE" | sed 's/[\/:@#]/_/g').sqsh"
FRAMEWORK_SUFFIX=$([[ "$FRAMEWORK" == "trt" ]] && printf '_trt' || printf '')
SPEC_SUFFIX=$([[ "$SPEC_DECODING" == "mtp" ]] && printf '_mtp' || printf '')
SPEC_SUFFIX=$([[ "$SPEC_DECODING" == "mtp" || "$SPEC_DECODING" == "draft_model" ]] && printf '_mtp' || printf '')
# Prefer a framework-tagged script (e.g. dsv4_fp4_b200_vllm.sh) so models
# with multiple inference engines can coexist; fall back to the historical
# name without an engine suffix (`_trt` for trt, bare for everyone else).
Expand Down