Repository navigation
fix: acceptance_rate per generate() call; acceptance figures from step counts - #58
Merged
Merged
Conversation
SpeculativeEngine created its AcceptanceTracker once and never reset it, so GenerationResult.acceptance_rate was a running mean over every earlier generate() call on the engine, warm-ups included, although the field is documented as the rate for this call. The tracker now resets at the start of each call, on both the batch-1 incremental path and the general step loop. Throughput, outputs, the bandit's updates and arm_steps were not affected. The new test makes several calls on one engine (batch 1 and 2, greedy and sampled) and checks each rate against that call's own arm_steps counts; it fails on main (3/3) and passes here.
acceptance_rate_mean in result files from 0.3.1 and earlier averages the engine's running means, so the README tables now show accepted / proposed draft tokens over the timed calls, computed from the stored arm_report counts by the new benchmarks.arm_report.pooled_acceptance (n/a without them). The table notes say this cell is computed, not stored. README regenerated with --readme-commit 3c43e08: only the acceptance cells and the two notes change. The unpacked result files are byte-identical.
…as a lesson Figure 9 plots accepted / proposed from each arm's stored arm_report counts (Qwen g2 0.95, lookup g6 0.54; throughput unchanged), and the caption follows. The qwen-g2 label moves inside the plot and the subtitle no longer runs past the right edge. Lesson 7 gains the acceptance tracker as another path no test had run (two calls on one engine). docs/benchmarks.md notes that its older acceptance values cannot be recomputed.
…s under Unreleased
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GenerationResult.acceptance_rateis documented as the rate "for this call", but the engine created itsAcceptanceTrackeronce (SpeculativeEngine.__init__) and never reset it. Every call after the first reported arunning mean over all earlier calls on that engine, warm-ups included. This PR resets the tracker per call and
recomputes the published acceptance figures from the per-step counts the result files already store. No GPU run is
needed.
How it was found
A step-by-step trace recorded on a T4 for a web replay (
flashspec_trace_t4.py, 0.3.1, 2 warm-ups and 3 repeats perprompt) reported acceptance values that did not match the trace's own steps. Replaying the trace's call order through
a running tracker reproduces all 20 reported values exactly. For example, HumanEval/1:rename reported 0.3075, but its
own steps give 0.8790.
What is affected
GenerationResult.acceptance_rateon every call after the first; the harness'sacceptance_rate_mean(a mean of those values); the README acceptance column; Figure 9 in
docs/lessons.md.update()gets eachstep's own counts),
arm_stepsandarm_report.Across the 100 result files that store both values,
acceptance_rate_meandiffers from accepted / proposed drafttokens (from
arm_report) by a median of 0.0126 and at most 0.0913. The stored value is higher in 40 files andlower in 60:
The 0.2.0 run (
8d036d7) and older files have noarm_report, so they cannot be recomputed.docs/benchmarks.mdsays so next to its historical tables.
Changes
fix(engine):self._acceptance_tracker.reset()at the start of both generate paths (batch-1 incremental andthe general step loop), and a sharper
acceptance_ratedocstring. Newtests/unit/test_acceptance_per_call.py:several calls on one engine at batch 1 and 2, greedy and sampled. Each rate must equal that call's own
accepted / proposed from
arm_steps.fix(readme):benchmarks.arm_report.pooled_acceptance(report)returns accepted / proposed summed over arms.scripts/ingest_gpu_bundle.pyuses it for the acceptance cell in both tables, orn/awithoutarm_report. Thetable notes say this cell is computed, not stored. README regenerated with
--readme-commit 3c43e08.docs(lessons): Figure 9 usespooled_acceptance. The caption changes from 0.94 to 0.95 and from 0.45 to 0.54;161 tok/s and 70 tok/s are unchanged. The qwen-g2 label moved inside the plot, and the subtitle, which was cut off
at the right edge, is shorter. Lesson 7 adds this bug: no test had made two calls in a row on one engine.
[Unreleased]→ Fixed.Before / after: README acceptance (bundle
3c43e08, every other cell unchanged)acceptance_rate_mean)arm_reportcounts)The full table's
flashspec_compiledrows show the same before and after values as the status table.Gate
Local, on this branch:
make lint(ruff, format, import-linter: 2 contracts kept),mypy --strict flashspec: cleanpytest tests/unit tests/integration -m "not gpu": 574 passed, 5 skipped (coverage 88.94%), includingtest_output_distribution.pypytest tests/chaos: 5 passedmain(3/3) and passes herepython scripts/readme_api.py --check: currentscripts/article/make_030_figures.py: regenerated figures match (test_article_figures.py)