[PowerX] add validated P75 and P90 GPU power metrics / 新增经过验证的 GPU P75 和 P90 功耗指标

[edwingao28]

Description

Emit measured P75 and P90 GPU power from the same validated telemetry/window used for average power. Sum the linearly interpolated device curves first, compute the time-weighted fleet percentiles, and divide by the participating GPU count for p75_power_w and p90_power_w. Both single-node SMI and multinode DCGM paths emit the fields; invalid windows withhold them.

This changes offline result processing only. It does not change benchmark execution, recipes, or performance. Existing average and energy calculations remain unchanged. Older averages cannot supply percentiles; the app changes replay original raw traces for existing points.

Related Issue

PowerX measured-power request. Companion P75 app PR: SemiAnalysisAI/InferenceX-app#1119. P90 app support landed in SemiAnalysisAI/InferenceX-app#1116.

Type of Change

• Bug fix
• New feature
• Configuration change
• Documentation update
• Other (please describe)

Checklist

• I have tested my changes locally
• I have updated documentation if necessary
• For every change that can affect benchmark performance and every recipe addition or modification, I have appended a new entry to the physical end of perf-changelog.yaml and have not edited historical entries — Not applicable: offline postprocessing only.
• Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /reuse-sweep-run on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one. — No GPU sweep requested for offline postprocessing; merge exception requires repository review.

Validation: 296 tests passed using the process-result workflow test command, including single/multinode integration and AgentX adapters; 34 original Qwen3.5 FP8 8k/1k audits replay with mean-power agreement within 0.000493 W. An independent uniformly timed 100001-point numerical oracle agrees with P75 within 0.00557 W and P90 within 0.01555 W. All original source hashes match, and adding P75 leaves existing P90 values unchanged. Tests cover asynchronous samples, opposing device ramps, time weighting, window clipping, and invalid-data withholding.

中文说明

在平均功耗所用的已验证遥测和测量窗口内，新增 GPU 实测功耗 P75 和 P90。先对各设备功耗曲线进行线性插值并按时间对齐求和，再计算按时间加权的整组 GPU P75 和 P90；p75_power_w 和 p90_power_w 分别将对应分位数除以参与测量的 GPU 数量。单节点 SMI 和多节点 DCGM 路径都输出这些指标，窗口验证失败时不发布。

本次仅修改离线结果处理，不改变基准测试执行、配方或性能，原有平均功耗和能耗计算保持不变。旧的平均值无法推算这些分位数。P90 应用支持已通过 InferenceX-app #1116 合并；P75 配套应用 PR SemiAnalysisAI/InferenceX-app#1119 重放同一批原始遥测，为现有数据补充 P75。

已同步英文和中文文档，process-result 工作流覆盖的单节点、多节点和 AgentX 路径共 296 项测试通过。重放 34 个 Qwen3.5 FP8 8k/1k 原始审计包后，平均功耗与原值的最大差异为 0.000493 W；独立的 100001 点等时间间隔数值校验与 P75、P90 的最大差异分别小于 0.00557 W 和 0.01555 W。原始文件哈希全部一致，新增 P75 后已有 P90 值保持不变。此离线处理改动未触发 GPU sweep，合并例外仍需遵循仓库审核规则。

---

Note

Medium Risk
Changes published benchmark metric schema and percentile semantics consumers must interpret correctly, but scope is offline post-processing with fail-closed validation and no runtime benchmark impact.

Overview
Adds time-weighted P75 and P90 measured GPU power to offline power aggregation for validated single-node SMI and multinode DCGM results, alongside existing average power and energy metrics.

A shared _percentile_total_power helper linearly aligns per-GPU curves, sums fleet draw, then computes duration-weighted percentiles over the same formal benchmark window used for energy integration. Aggregates gain p75_total_gpu_power_w, p90_total_gpu_power_w, and per-GPU p75_power_w / p90_power_w (fleet percentile ÷ GPU count—not per-device percentiles). Invalid telemetry withholds all four fields; validation sidecars record power_percentile_method. English and Chinese ingestion docs describe the contract; tests cover opposing ramps, async samples, window clipping, and invalid-data withholding.

Benchmark execution and existing mean/energy math are unchanged; historical rows need raw trace replay for the new fields.

^{Reviewed by Cursor Bugbot for commit 4b70d21. Bugbot is set up for automated code reviews on this repo. Configure here.}

[claude]

I reviewed this PR and didn't find any bugs. Because it adds a new time-weighted percentile algorithm (bisection over piecewise-linear-interpolated, summed per-device power curves) that feeds production power metrics across both single-node and multinode paths, a human look would still be worthwhile.

What was reviewed: the new _p90_total_power bisection/slope-accumulation logic in common.py and its wiring into single_node.py/multinode.py; the withholding gate that ties p90_* to the same validity reasons as the existing energy metrics; the four new test_aggregate_power.py cases (hand-verified the 140 W clipped-ramp and 300/600 W opposing-ramp/async-sample expectations by direct calculation); and the multinode ramp test's new 1456.0/364.0 assertions (independently re-derived from the fixture's linear ramp, not copied from the implementation). Also checked whether skipping a POWER_METRIC_SCHEMA_VERSION bump for the two new WHOLE_METRIC_KEYS entries is safe — the documented schema-version contract only governs joules_per_* whole-vs-role-scoped semantics, which are unaffected here, so no bump appears required.

Extended reasoning...

Overview

The PR adds two new derived power metrics (p90_total_gpu_power_w and p90_power_w) computed via a new _p90_total_power helper in infx/results/power/common.py, wired into both the single-node (SMI) and multinode (DCGM) validation/integration paths, plus matching bilingual docs and new unit tests. The core algorithm sums per-device piecewise-linear power curves onto a shared timeline (recording slope changes at each device's own sample knots), then bisects to find the time-weighted 90th percentile of the summed curve over the validated benchmark window.

Security risks

None. This is offline, read-only post-processing of already-validated telemetry CSVs; no new I/O surface, no untrusted input parsing beyond what already existed, no auth/crypto/permission changes.

Level of scrutiny

This warrants a solid human read despite no bugs found: it's a new numerical algorithm (bisection over a synthesized piecewise-linear step function) touching a production metrics/schema surface (sidecar payload, WHOLE_METRIC_KEYS) across two code paths (single-node and multinode) rather than a small mechanical change. I traced the gating logic carefully: in single_node.py, p90_total is computed only if not reasons (i.e., exactly when the integration is fully valid), matching the guard already used for avg_power_w/per_gpu_energy_j completeness, so _derived_metrics's ValueError guard for None p90 fields can never fire on a power_valid result. In multinode.py, per_key_samples[device.key] is indexed unconditionally (no .get fallback) for every expected_devices entry when building the p90 input; I verified this can't KeyError because the code returns early whenever audit.reasons is non-empty, and _validate_devices already appends expected_device_missing (which propagates to audit.reasons) for any expected device absent from observed, which itself is derived directly from the same rows/per_key_samples — so every expected device is guaranteed present in per_key_samples by the time it's indexed.

I also specifically checked the new function for a division-by-zero risk on duplicate/near-duplicate timestamps within a single device's samples (since, unlike the pre-existing trapezoid _integrate_device, the new function does divide by (right_t - left_t) when computing slopes). Tracing read_samples's timestamp_non_monotonic check (current.timestamp_unix <= previous.timestamp_unix over scrape_seq-sorted rows), I confirmed by a short monotonicity argument that any duplicate timestamp for the same device necessarily produces an adjacent equal pair in scrape_seq order and is therefore always rejected before reaching the p90 computation — so this isn't a live risk in the current diff.

Other factors

The four new unit tests in utils/test_aggregate_power.py and the two new assertions in utils/test_aggregate_power_multinode.py are genuinely independently derived rather than copied from the algorithm: I re-derived the 140 W clipped-window value, the 300/600 W opposing-ramp and async-sample values, and the 1456.0/364.0 multinode-ramp values by hand from the fixtures' underlying (mostly affine) power functions, and all matched the asserted expectations, which supports the PR's stated testing rigor. I also considered whether adding two keys to WHOLE_METRIC_KEYS without bumping POWER_METRIC_SCHEMA_VERSION conflicts with the documented schema contract in .github/AGENT_OPERATIONS.md; that contract's stated scope is specifically about joules_per_* whole-vs-role-scoped comparability, which these new keys don't touch, so I did not treat this as a blocking concern, though it's a judgment call. Given the algorithm's non-trivial numerical construction and its reach into a shared metrics schema, I'm deferring rather than approving even though I found nothing wrong.