fix(speedbench-al): auto-PR commits golden AL matrix to golden_al_distribution/ - #2847
Draft
connectors-testing-pplx wants to merge 1 commit into
Conversation
…tribution/ The speedbench-al.yml open-pr step copied the measured AL matrix to benchmarks/speedbench-reference-al.yaml, a path that does not exist on main and is not consumed by anything. The canonical golden AL curves live in golden_al_distribution/<model>_<method>.yaml and are referenced from the README table, the AgentX methodology article, and the agentic throughput recipes (SGLANG_SIMULATE_ACC_LEN etc.). Because the auto-PR never landed the matrix in the right place, every golden YAML to date (qwen3.8next included) had to be committed by hand, with the source-run URL header pasted in separately. Fix the step so a completed collection run commits directly: - Resolve the target filename as <model-prefix>_<spec-decoding>.yaml with an explicit golden-yaml-name input override for non-standard suffixes (e.g. the dspark probabilistic-sample curves). - Prepend the '# Source GitHub Actions run:' header the collector does not emit, so the committed file matches the canonical shape of the other curves. - Update the README 'Current golden curves' table row in place (replace if the file already has one, insert before 'Primary references' otherwise). - Rewrite the PR title/body to describe the golden_al_distribution/ commit. This unblocks a push-button qwen3.8-flash-next collection (and every other model) from landing its AL distribution automatically instead of requiring a manual copy from the workflow artifact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
speedbench-al.ymlworkflow'sopen-prstep copies the measured AL matrix tobenchmarks/speedbench-reference-al.yaml— a path that does not exist on main and is not consumed by anything. The canonical golden AL curves live ingolden_al_distribution/and are referenced from:packages/app/src/components/datasets/agentx-methodology-article.tsx),benchmarks/single_node/agentic/qwen3.8next_fp4_b300_sglang_mtp.sh→SGLANG_SIMULATE_ACC_LEN=2.32fromgolden_al_distribution/qwen3.8next_mtp.yaml).Because the auto-PR never landed the matrix in the right place, every golden YAML to date (qwen3.8-flash-next included) had to be committed by hand, with the
# Source GitHub Actions run:header pasted in separately. This is the friction blocking a complete qwen3.8-flash-next AL distribution: the only successful run (33034290269) collectedthinking=[on],mtp=[1 2 3 4 5 6]and was committed manually as a partialqwen3.8next_mtp.yaml.Fix
Rewrite the
open-prstep so a completed collection run commits directly:golden_al_distribution/<model-prefix>_<spec-decoding>.yamlwith a newgolden-yaml-nameinput override for non-standard suffixes.# Source GitHub Actions run:line the collector does not emit, matching the canonical shape of the other curves.golden_al_distribution/commit and link the source run.Why not also add the missing thinking_off / mtp 7-8 cells for qwen3.8-flash-next?
Those cells require a GPU collection run on a B300 runner (the
speedbench-al.ymlworkflow). I have onlypullaccess to this org repo, so I cannot dispatch the workflow, and I will not fabricate acceptance-length values — every golden AL is a real SPEED-Bench measurement with an auditable Actions run behind it. This PR removes the manual-copy step so the next dispatch (thinking-modes=off on,mtp-list=1 2 3 4 5 6 7 8) lands the complete qwen3.8-flash-next matrix automatically.Verification
yaml.safe_loadon the workflow → parses cleanly.golden_al_distribution/README.md: matches and replaces the existingqwen3.8next_mtp.yamlrow.Checklist
.github/workflows/speedbench-al.ymlchanged