feat: add controlled AgentX power experiment / 添加受控 AgentX 功耗实验 - #2827
feat: add controlled AgentX power experiment / 添加受控 AgentX 功耗实验#2827edwingao28 wants to merge 3 commits into
Conversation
为 B200、B300 和 MI355X 添加固定版本的 Qwen FP8 TP4 AgentX 功耗实验，关闭推测解码与 CPU offload。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
关联功耗实验草稿 PR。
修复 PowerX 验证使用的 Enroot digest 导入语法，并为 NVIDIA 单节点任务挂载持久化 HF 缓存。记录模型缓存预检查并补充回归测试。
|
Claude finished @edwingao28's task in 10m 51s —— View job PR Review — controlled AgentX PowerX experiment
SummaryFound 1 blocking issue (posted inline): the shared Everything else checked out:
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| local image="$1" | ||
| local image_without_digest="$image" |
There was a problem hiding this comment.
🔴 BLOCKING: enroot_uri_for_image mangles image references that are already written in Enroot's REGISTRY#IMAGE syntax, and this PR newly routes all single-node cluster:b200-nscale imports through it.
For nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc14, first_component="${image_without_digest%%/*}" yields nvcr.io#nvidia (contains a .), so registry="nvcr.io#nvidia" and the function prints docker://nvcr.io#nvidia#tensorrt-llm/release:1.3.0rc14 — a double-# URI that fails Enroot's own consumer grammar (registry is [^#]+, image charset excludes #; the very regex quoted in utils/test_container_utils.py).
Why it matters: configs/nvidia-master.yaml has nine current configs with #-style images on cluster:b200-nscale (e.g. dsr1-fp4-b200-trt, dsr1-fp8-b200-trt, qwen3.5-fp4-b200-trt with nvcr.io#nvidia/tensorrt-llm/release:*, and the ghcr.io#semianalysisai/trtllm-deepseek-v4 dsv4 configs). Before this PR they imported via docker://$IMAGE verbatim, which is valid Enroot syntax and worked. The failure is currently masked wherever the cached .sqsh already exists, but any fresh import (cache eviction, new node, image bump) will fail at enroot import. The same latent issue now applies to launch_mi355x-amds.sh and launch_b300-dsxe.sh (no # images target them today), and pre-existed in launch_gb200-nv.sh.
Fix: normalize the # form to slash form before parsing — this reproduces the exact previously-working URI for tag-only refs and makes digest pinning work for those registries too:
| local image="$1" | |
| local image_without_digest="$image" | |
| local image="$1" | |
| # Normalize Enroot's REGISTRY#IMAGE syntax (e.g. nvcr.io#nvidia/..., | |
| # used by existing master-config images) to slash form before parsing. | |
| image="${image/\#//}" | |
| local image_without_digest="$image" |
Please also add regression cases to utils/test_container_utils.py, e.g. ("nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc14", "nvcr.io", "nvidia/tensorrt-llm/release", "1.3.0rc14") and (f"ghcr.io#org/image:tag@{DIGEST}", "ghcr.io", "org/image", DIGEST).
| first_component="${image_without_digest%%/*}" | ||
| if [[ "$image_without_digest" == */* && ( "$first_component" == *.* || "$first_component" == *:* || "$first_component" == "localhost" ) ]]; then | ||
| registry="$first_component" | ||
| repository="${image_without_digest#*/}" | ||
| else | ||
| registry="registry-1.docker.io" | ||
| repository="$image_without_digest" | ||
| fi |
There was a problem hiding this comment.
🔴 enroot_uri_for_image() splits on the first '/' without checking for a pre-existing '#' registry separator, so images already written in Enroot's registry#repo:tag form (e.g. nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc14, used by dsr1-fp4-b200-trt, dsr1-fp8-b200-trt(-mtp), qwen3.5-fp4-b200-trt(-mtp) on cluster:b200-nscale) get mangled into docker://nvcr.io#nvidia#tensorrt-llm/release:1.3.0rc14, breaking enroot import for recipes that worked before this diff (IMAGE previously passed through verbatim). Fix: detect an existing '#' and split registry/repository there first, before falling back to slash-based custom-registry detection. [also at: runners/launch_b300-dsxe.sh:217 - import_squash_image() now runs every image_ref through enroot_uri_for_image, but the DCGM exporter call site at line…]
Extended reasoning...
configs/nvidia-master.yaml pre-encodes several active recipes (e.g. dsr1-fp4-b200-trt at line 1052, runner cluster:b200-nscale, no digest) as image: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc14 — the manual workaround this codebase already used for custom-registry, non-digest images before enroot_uri_for_image existed. Before this diff, launch_b200-nscale-compat.sh's single-node branch (IS_MULTINODE=false, lines ~570-600) did enroot import -o "$SQUASH_FILE" docker://$IMAGE, passing that string through unchanged and correctly. After this diff it now calls ENROOT_IMAGE_URI=$(enroot_uri_for_image "$IMAGE") first. In enroot_uri_for_image, first_component="${image_without_digest%%/*}" takes everything up to the FIRST '/', which for this string is "nvcr.io#nvidia" (the '#' occurs before the '/'). Since that contains a '.', it is classified as a custom registry: registry="nvcr.io#nvidia", repository="tensorrt-llm/release:1.3.0rc14". With no digest and registry != registry-1.docker.io, it prints docker://nvcr.io#nvidia#tensorrt-llm/release:1.3.0rc14 — a URI with two '#'…
Verification: normal. enroot_uri_for_image() (runners/container_utils.sh) assumes its input is in Docker / syntax and never detects input that is already in Enroot registry#repo:tag form. Tracing the non-digest branch with the real config value nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc14: - line 17 first_component="${image_without_digest%%/*}" splits on the first /, yielding nvcr.io#nvidia. -…
1 participant
Adds Qwen3.5 FP8 AgentX power configurations for B200, B300 and MI355X: four GPUs, TP4/EP1, native decoding, prefix caching, no CPU KV offload, and candidate session concurrency 1/4/12. Model/tokenizer and container revisions are pinned; recipes verify the trace revision and require valid GPU power with zero profiling request errors.
The initial B200 qualification failed before model startup because Enroot treated Docker's tag@digest syntax as authentication. The launchers now reuse the existing GB200 manifest-reference converter, and B200/B300 explicitly mount persistent HF caches for pinned snapshots. Eight converter regression cases pass. The existing 88 power/AgentX tests, nine-row matrix generation, Bash syntax, whitespace, and changelog validation also pass.
Hardware qualification is in progress at https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33921508108. This draft contains no scored measurements. The proposed 27 one-hour measurements remain gated on power, serving-GPU identity, replay coverage, and quality checks.
为 B200、B300 和 MI355X 添加 Qwen3.5 FP8 AgentX 功耗配置：四张 GPU、TP4/EP1、原生解码、前缀缓存、无 CPU KV offload，候选会话并发为 1/4/12。固定模型、tokenizer 和镜像版本，检查 trace 数据集版本，并要求有效 GPU 功耗和零 profiling 请求错误。
首轮 B200 验证在模型启动前失败：Enroot 将 Docker 的 tag@digest 语法解释为认证信息。现复用 GB200 已有的 manifest 引用转换逻辑，并为 B200/B300 显式挂载持久化 HF 缓存。8 项转换回归测试、88 项功耗测试、九个矩阵条目、Bash 语法、空白及 changelog 验证均通过。
硬件验证正在上述 CI 中运行，尚无正式测量结果。后续 27 次一小时测量须先通过功耗、实际服务 GPU 身份、回放覆盖率和质量检查。
Note
Medium Risk
Changes Slurm/Enroot image import and container mount paths on production benchmark runners; misconfiguration could break unrelated jobs, though scope is launcher and new opt-in config keys rather than core serving logic.
Overview
Adds controlled AgentX “PowerX” matrix keys for B200, B300, and MI355X (
qwen3.5-fp8-*-sglang-agentic-powerx): TP4/EP1, native decoding, no CPU KV offload, pinned Weka trace corpus, zero allowed replay errors, mandatory GPU power andsglang:server metrics. New SGLang launcher recipes download a pinned HF model revision, setAIPERF_TOKENIZERto that snapshot, and record runtime/cache provenance for qualification runs.Agentic replay now passes
--tokenizer ${AIPERF_TOKENIZER:-$MODEL}so tokenizer loading stays aligned with the served checkpoint when wire names differ from HF ids.Cluster runners fix digest-pinned Docker images for Enroot 3.x by centralizing
enroot_uri_for_imageinrunners/container_utils.sh(with pytest coverage) and using it on B200/B300/MI355X/GB200 launch paths. B200 and B300 single-node jobs also mount persistent Hugging Face hub caches and log model download dry-runs before weight load.Docs (EN/ZH) and
perf-changelog.yamldocument the PowerX keys and runner behavior; B200/B300 bench scripts delegate to a shared Blackwell PowerX launcher.Reviewed by Cursor Bugbot for commit 1c0103f. Bugbot is set up for automated code reviews on this repo. Configure here.