Skip to content

fix(speedbench-al): auto-PR commits golden AL matrix to golden_al_distribution/ - #2847

Draft
connectors-testing-pplx wants to merge 1 commit into
SemiAnalysisAI:mainfrom
connectors-testing-pplx:fix/speedbench-al-golden-distribution-autopr
Draft

fix(speedbench-al): auto-PR commits golden AL matrix to golden_al_distribution/#2847
connectors-testing-pplx wants to merge 1 commit into
SemiAnalysisAI:mainfrom
connectors-testing-pplx:fix/speedbench-al-golden-distribution-autopr

Conversation

@connectors-testing-pplx

Copy link
Copy Markdown

Problem

The speedbench-al.yml workflow's open-pr step copies the measured AL matrix to benchmarks/speedbench-reference-al.yaml — a path that does not exist on main and is not consumed by anything. The canonical golden AL curves live in golden_al_distribution/ and are referenced from:

  • the README "Current golden curves" table,
  • the AgentX methodology article (packages/app/src/components/datasets/agentx-methodology-article.tsx),
  • the agentic throughput recipes (e.g. benchmarks/single_node/agentic/qwen3.8next_fp4_b300_sglang_mtp.shSGLANG_SIMULATE_ACC_LEN=2.32 from golden_al_distribution/qwen3.8next_mtp.yaml).

Because the auto-PR never landed the matrix in the right place, every golden YAML to date (qwen3.8-flash-next included) had to be committed by hand, with the # Source GitHub Actions run: header pasted in separately. This is the friction blocking a complete qwen3.8-flash-next AL distribution: the only successful run (33034290269) collected thinking=[on], mtp=[1 2 3 4 5 6] and was committed manually as a partial qwen3.8next_mtp.yaml.

Fix

Rewrite the open-pr step so a completed collection run commits directly:

  1. Target filename: resolve golden_al_distribution/<model-prefix>_<spec-decoding>.yaml with a new golden-yaml-name input override for non-standard suffixes.
  2. Source-run header: prepend the # Source GitHub Actions run: line the collector does not emit, matching the canonical shape of the other curves.
  3. README table: update the "Current golden curves" row in place (regex replace if the file already has one; insert before "## Primary references" otherwise).
  4. PR title/body: describe the golden_al_distribution/ commit and link the source run.

Why not also add the missing thinking_off / mtp 7-8 cells for qwen3.8-flash-next?

Those cells require a GPU collection run on a B300 runner (the speedbench-al.yml workflow). I have only pull access to this org repo, so I cannot dispatch the workflow, and I will not fabricate acceptance-length values — every golden AL is a real SPEED-Bench measurement with an auditable Actions run behind it. This PR removes the manual-copy step so the next dispatch (thinking-modes=off on, mtp-list=1 2 3 4 5 6 7 8) lands the complete qwen3.8-flash-next matrix automatically.

Verification

  • yaml.safe_load on the workflow → parses cleanly.
  • README-row regex tested against the real golden_al_distribution/README.md: matches and replaces the existing qwen3.8next_mtp.yaml row.
  • Collector script and partial YAML unchanged; only the auto-PR plumbing changes.

Checklist

  • Workflow YAML validates
  • Only .github/workflows/speedbench-al.yml changed
  • No fabricated AL values

…tribution/

The speedbench-al.yml open-pr step copied the measured AL matrix to
benchmarks/speedbench-reference-al.yaml, a path that does not exist on main and
is not consumed by anything. The canonical golden AL curves live in
golden_al_distribution/<model>_<method>.yaml and are referenced from the README
table, the AgentX methodology article, and the agentic throughput recipes
(SGLANG_SIMULATE_ACC_LEN etc.). Because the auto-PR never landed the matrix in
the right place, every golden YAML to date (qwen3.8next included) had to be
committed by hand, with the source-run URL header pasted in separately.

Fix the step so a completed collection run commits directly:
- Resolve the target filename as <model-prefix>_<spec-decoding>.yaml with an
  explicit golden-yaml-name input override for non-standard suffixes (e.g. the
  dspark probabilistic-sample curves).
- Prepend the '# Source GitHub Actions run:' header the collector does not
  emit, so the committed file matches the canonical shape of the other curves.
- Update the README 'Current golden curves' table row in place (replace if the
  file already has one, insert before 'Primary references' otherwise).
- Rewrite the PR title/body to describe the golden_al_distribution/ commit.

This unblocks a push-button qwen3.8-flash-next collection (and every other
model) from landing its AL distribution automatically instead of requiring a
manual copy from the workflow artifact.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant