Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion benchmarks/single_node/agentic/kimik3_fp4_b300_vllm_mtp.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ set -x
# Agentic trace replay for Kimi-K3 (MXFP4) on B300: TP8 x DCP8, TokenspeedMLA,
# Mooncake as the external KV tier. Concurrency selects the arm:
# conc <= 8 DSpark level 7, golden AL 3.84
# conc 16 DSpark level 3, golden AL 3.00
# conc 10-16 DSpark level 3, golden AL 3.00
# conc > 16 no drafting
# Keep the arms disjoint in concurrency: exp-name carries conc and spec but not
# the arm, so a shared concurrency would collide.
Expand Down
6 changes: 3 additions & 3 deletions configs/nvidia-master.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1489,7 +1489,7 @@ kimik3-fp4-b300-vllm-agentic-dspark:
image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-5894fdf
model: moonshotai/Kimi-K3
model-prefix: kimik3
runner: cluster:b300-nv
runner: cluster:b300-dsxe
precision: fp4
framework: vllm
multinode: false
Expand All @@ -1502,9 +1502,9 @@ kimik3-fp4-b300-vllm-agentic-dspark:
search-space:
# TP8-only: a ~1.5 TB MXFP4 checkpoint does not fit below 8 GPUs.
# One entry for every arm: the recipe drafts at DSpark 7 up to conc 8,
# DSpark 3 at conc 16, and not at all above. Keep the concurrencies
# DSpark 3 at conc 10-16, and not at all above. Keep the concurrencies
# disjoint across arms so exp-names stay unique.
- { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 70] }
- { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [1, 2, 4, 8, 10, 12, 14, 16, 24, 32, 40, 48, 56, 70] }

dsr1-fp8-b200-trt:
image: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc14
Expand Down
9 changes: 9 additions & 0 deletions perf-changelog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6902,3 +6902,12 @@
- "Lower mem-fraction-static from 0.89 to 0.86 on every arm: TP4, TP8 and TP8 with DP attention. swa-full-tokens-ratio becomes per-arm: 0.10 on the tensor-parallel arms as before, 0.15 under DP attention."
- "Drop concurrency 2 and 10 from the TP4 arm, leaving [1, 4, 8], and drop concurrency 16 from the TP8 hicache arm, leaving [32, 48]. Concurrency 16 remains on the TP8 no-offload arm."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2800

- config-keys:
- kimik3-fp4-b300-vllm-agentic-dspark
scenario-type:
- agentic-coding
description:
- "Add concurrency 10, 12 and 14 to the TP8 x DCP8 Mooncake sweep, so the conc-list becomes [1, 2, 4, 8, 10, 12, 14, 16, 24, 32, 40, 48, 56, 70]. The new points fall in the recipe's existing conc 9-16 band and therefore draft with DSpark level 3 at the committed golden AL 3.00, the same arm as concurrency 16."
- "Repoint the runner from cluster:b300-nv, whose launcher and runner labels were retired in #2826, to cluster:b300-dsxe so the sweep can schedule."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2846
Loading