Skip to content

fix(agentx): refresh GB200 Dynamo-vLLM with complete DP metrics #2822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cquil11 wants to merge 5 commits into
base: main
Choose a base branch
from
Open

fix(agentx): refresh GB200 Dynamo-vLLM with complete DP metrics #2822

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f712f55
fix(agentx): refresh GB200 Dynamo-vLLM with complete DP metrics
cquil11 Sep 4, 2026
93b1a33
chore: link metrics refresh changelog to PR 2822
cquil11 Sep 4, 2026
aaefa04
fix(agentx): allow c512 warmup and metric export to finish
cquil11 Sep 4, 2026
5c6e594
fix(agentx): allow backpressured GB200 warmup uploads
cquil11 Sep 4, 2026
b742758
fix(agentx): reduce prefill activation fragmentation
cquil11 Sep 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions ...e/srt-slurm-recipes/vllm/deepseek-v4/agentic/disagg-gb200-1p1d-dep8-dep8-mtp-agentic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ backend:
type: vllm
connector: null
dp_launch_mode: per_node
prefill_environment: &worker_environment
decode_environment: &worker_environment
HF_HUB_CACHE: /hf_hub_cache
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
VLLM_RPC_TIMEOUT: "600000"
Expand All @@ -66,7 +66,9 @@ backend:
UCX_NET_DEVICES: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1"
UCX_TLS: "rc,cuda_copy"
NCCL_IB_HCA: "mlx5_0,mlx5_1,mlx5_2,mlx5_3"
decode_environment: *worker_environment
prefill_environment:
<<: *worker_environment
PYTORCH_CUDA_ALLOC_CONF: "expandable_segments:True"

@cursor cursor Bot Sep 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NixlConnector conflicts with expandable segments

High Severity

Prefill now sets PYTORCH_CUDA_ALLOC_CONF to expandable_segments:True while both workers still use NixlConnector for KV transfer. vLLM treats that pairing as incompatible because expandable CUDA VMM can remap pages that NIXL already registered for RDMA, so prefill either fails at startup or the disagg KV path can corrupt or stall.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit b742758. Configure here.

vllm_config:
prefill: &dep8_config
kv-transfer-config: '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
Expand Down Expand Up @@ -116,7 +118,8 @@ benchmark:
IS_MULTINODE: "true"
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"
AIPERF_HTTP_TCP_USER_TIMEOUT: "900000"
AIPERF_DATASET_MMAP_CACHE_DIR: /aiperf_mmap_cache
HF_HUB_CACHE: /hf_hub_cache
WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:"
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:kv_cache_usage_perc"
11 changes: 7 additions & 4 deletions ...e/srt-slurm-recipes/vllm/deepseek-v4/agentic/disagg-gb200-2p1d-dep8-dep8-mtp-agentic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ identity:
dynamo: { version: "1.3.1", install: true }
setup_script: vllm-container-deps.sh
environment: { ETCD_LEASE_TTL: "7200" }
slurm: { time_limit: "8:00:00" }
slurm: { time_limit: "12:00:00" }
health_check: { max_attempts: 2160, interval_seconds: 10 }

resources:
Expand Down Expand Up @@ -45,7 +45,7 @@ backend:
type: vllm
connector: null
dp_launch_mode: per_node
prefill_environment: &worker_environment
decode_environment: &worker_environment
HF_HUB_CACHE: /hf_hub_cache
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
VLLM_RPC_TIMEOUT: "600000"
Expand All @@ -61,7 +61,9 @@ backend:
UCX_NET_DEVICES: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1"
UCX_TLS: "rc,cuda_copy"
NCCL_IB_HCA: "mlx5_0,mlx5_1,mlx5_2,mlx5_3"
decode_environment: *worker_environment
prefill_environment:
<<: *worker_environment
PYTORCH_CUDA_ALLOC_CONF: "expandable_segments:True"
vllm_config:
prefill: &dep8_config
kv-transfer-config: '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
Expand Down Expand Up @@ -111,7 +113,8 @@ benchmark:
IS_MULTINODE: "true"
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"
AIPERF_HTTP_TCP_USER_TIMEOUT: "900000"
AIPERF_DATASET_MMAP_CACHE_DIR: /aiperf_mmap_cache
HF_HUB_CACHE: /hf_hub_cache
WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:"
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:kv_cache_usage_perc"
2 changes: 2 additions & 0 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7978,6 +7978,7 @@ dsv4-fp4-gb200-dynamo-vllm-agentic-mtp-disagg:
- search-space:
# One two-tray DEP8 prefill and decode worker covers the balanced region.
- spec-decoding: mtp
kv-offloading: none
conc-list: [32, 64, 96, 128, 192, 256]
prefill:
num-worker: 1
Expand All @@ -7995,6 +7996,7 @@ dsv4-fp4-gb200-dynamo-vllm-agentic-mtp-disagg:
dp-attn: true
# A second DEP8 prefill worker extends the high-concurrency region.
- spec-decoding: mtp
kv-offloading: none
conc-list: [192, 256, 384, 512]
prefill:
num-worker: 2
Expand Down
8 changes: 8 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6868,3 +6868,11 @@
description:
- "Refresh the DeepSeek-V4-Pro FP4 B200 vLLM image and search space, pin the Nscale NVFP4 checkpoint, use the FlashInfer CuTeDSL MoE backend for expert-parallel runs with EPLB disabled, set GPU memory utilization to 0.95, and cap the model length at 12288."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2534

- config-keys:
- dsv4-fp4-gb200-dynamo-vllm-agentic-mtp-disagg
scenario-type:
- agentic-coding
description:
- "Refresh: we're refreshing this run with the correct metrics."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2822
8 changes: 8 additions & 0 deletions runners/launch_gb200-nv.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -420,6 +420,14 @@ elif [[ "$IS_AGENTIC" == "1" && "$MODEL_PREFIX" == "minimaxm3" && "$PRECISION" =
mkdir -p recipes/vllm/minimax-m3/gb200-fp4/agentic
cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/vllm/minimax-m3/gb200-fp4/agentic" \
recipes/vllm/minimax-m3/gb200-fp4/agentic
# Collect both nodes' DP metrics for resident DeepSeek-V4 disaggregation.
elif [[ "$IS_AGENTIC" == "1" && "$MODEL_PREFIX" == "dsv4" && "$PRECISION" == "fp4" && "$FRAMEWORK" == "dynamo-vllm" && "$DISAGG" == "true" && "$KV_OFFLOADING" == "none" ]]; then
git clone --branch fix/custom-vllm-dp-metrics --single-branch https://github.com/SemiAnalysisAI/srt-slurm.git "$SRT_REPO_DIR" || exit 1
cd "$SRT_REPO_DIR" || exit 1
git checkout bde8ca8a7a14e124531085cf0d908d3907b11ad6 || exit 1
mkdir -p recipes/vllm/deepseek-v4/agentic
cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic" \
recipes/vllm/deepseek-v4/agentic
# These AgentX submissions use released srt-slurm custom-benchmark metrics
# discovery so AIPerf receives every logical worker endpoint.
elif [[ "$IS_AGENTIC" == "1" && (( "$MODEL_PREFIX" == "qwen3.5" && "$PRECISION" == "fp4" && "$FRAMEWORK" == "dynamo-sglang" ) || ( "$MODEL_PREFIX" == "dsv4" && "$PRECISION" == "fp4" && "$FRAMEWORK" == "dynamo-vllm" )) ]]; then
Expand Down
Loading