Skip to content

perf(agentx): restore LMCache retrieves on Kimi-K3 MI355X DCP8 #2831

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sammshen wants to merge 2 commits into
base: main
Choose a base branch
from
Open

perf(agentx): restore LMCache retrieves on Kimi-K3 MI355X DCP8 #2831

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
565db2a
perf(agentx): restore LMCache retrieves on K3 MI355X DCP8
sammshen Sep 4, 2026
6d4b648
chore: fill changelog pr-link
sammshen Sep 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,8 +149,10 @@ case "${KV_OFFLOAD_BACKEND:-}" in
lmcache)
require_agentic_kv_offload_backend "$KV_OFFLOAD_BACKEND"

LMCACHE_VERSION=0.5.5.dev89+rocm7.2
LMCACHE_ROCM_INDEX="https://github.com/LMCache/LMCache/releases/expanded_assets/nightly-rocm"
# A versioned -rocm release, not nightly-rocm: that tag keeps only the
# newest wheel, so an exact .devN pin stops resolving at the next build.
LMCACHE_VERSION=0.5.5rc5+rocm7.2
LMCACHE_ROCM_INDEX="https://github.com/LMCache/LMCache/releases/expanded_assets/v0.5.5rc5-rocm"

agentic_pip_install --quiet --no-cache-dir --no-deps \
"sortedcontainers==2.4.0" \
Expand Down Expand Up @@ -190,8 +192,10 @@ case "${KV_OFFLOAD_BACKEND:-}" in
# so 3072 is the minimum valid chunk. The multi-group layout also
# requires one object group per sliding-window size:
# --separate-object-groups.
LMCACHE_PORT=6555
LMCACHE_HTTP_PORT=8090
# Derive from $PORT: the launcher assigns it per-runner so two cells
# sharing a host do not collide on the MP server.
LMCACHE_PORT=$((PORT + 13000))
LMCACHE_HTTP_PORT=$((PORT + 14000))
LMCACHE_LOG="$RESULT_DIR/lmcache_server.log"

LMCACHE_L1_SIZE_GB="$TOTAL_CPU_DRAM_GB"
Expand All @@ -216,6 +220,10 @@ case "${KV_OFFLOAD_BACKEND:-}" in
--http-port "$LMCACHE_HTTP_PORT"
--l1-size-gb "$LMCACHE_L1_SIZE_GB"
--l1-init-size-gb 10
# The L1 read lock is held from prefetch through retrieve; at these
# concurrencies a queued request exceeds the 300s default and the
# retrieve fails.
--l1-read-ttl-seconds 3600
--chunk-size "$LMCACHE_CHUNK_SIZE"
--separate-object-groups
--enable-extra-logging
Expand Down
4 changes: 2 additions & 2 deletions configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -637,8 +637,8 @@ kimik3-fp4-mi355x-vllm-agentic-mtp:
- dram-utilization: 0.60
search-space:
- { tp: 8, ep: 1, dcp-size: 1, kv-offloading: none, conc-list: [1] , spec-decoding: mtp}
- { tp: 8, ep: 1, dcp-size: 1, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.5.dev89+rocm7.2" }, conc-list: [4, 8, 10, 12, 14], spec-decoding: mtp }
- { tp: 8, ep: 1, dcp-size: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.5.dev89+rocm7.2" }, conc-list: [44, 48, 52], spec-decoding: mtp }
- { tp: 8, ep: 1, dcp-size: 1, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.5rc5+rocm7.2" }, conc-list: [4, 8, 10, 12, 14], spec-decoding: mtp }
- { tp: 8, ep: 1, dcp-size: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.5rc5+rocm7.2" }, conc-list: [44, 48, 52], spec-decoding: mtp }

# Kimi-K3 MXFP4 agentic-coding benchmark on MI355X via ATOM with DSpark
# speculative decoding. Acceptance is pinned to the committed golden curve in
Expand Down
11 changes: 11 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6887,3 +6887,14 @@
- "Bump the MI355X MiniMax-M3 AgentX image from vllm/vllm-openai-rocm:nightly-6d4562c59b97b4e35d459ff9389e71b6fe4995de to the latest immutable nightly, vllm/vllm-openai-rocm:nightly-8a728663c1c3eeace834a95f5654fa653cc1998c (digest sha256:0d07767c0b8471eaca0f1eca97899620dfb4ad33ff2f5226789afef1a877e9bc)."
- "Pick up the optimized BF16 MiniMax-M3 indexer and routed GEMM from vLLM PRs #54682 and #54845; use FULL_DECODE_ONLY CUDA graphs while breakable CUDA graphs remain disabled."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2825

- config-keys:
- kimik3-fp4-mi355x-vllm-agentic-mtp
scenario-type:
- agentic-coding
description:
- "Pin lmcache 0.5.5rc5+rocm7.2 from the versioned v0.5.5rc5-rocm release, replacing the nightly-rocm 0.5.5.dev89+rocm7.2 pin whose wheel that rolling tag no longer carries."
- "rc5 carries the DCP cache-identity fix (LMCache #4936): the scheduler and the workers derived different LMCache model names under DCP, so every lookup missed and the DRAM tier was write-only at concurrency 44-52 (120 GB/GPU stored, 0 retrieved, external hit rate 0.0%)."
- "Derive the LMCache MP server ports from $PORT rather than the fixed 6555/8090, so two cells sharing a runner cannot collide."
- "Raise the L1 read-lock TTL to 3600s: the lock is held from prefetch through retrieve, and a queued request at these concurrencies exceeds the 300s default and fails the retrieve."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2831
Loading