Skip to content

fix: acceptance_rate per generate() call; acceptance figures from step counts - #58

Merged
Mattral merged 4 commits into
mainfrom
fix/acceptance-per-call
Oct 29, 2026
Merged

Mattral merged 4 commits into
mainfrom
fix/acceptance-per-call

Conversation

@Mattral

@Mattral Mattral commented Oct 29, 2026

Copy link
Copy Markdown
Owner

GenerationResult.acceptance_rate is documented as the rate "for this call", but the engine created its
AcceptanceTracker once (SpeculativeEngine.__init__) and never reset it. Every call after the first reported a
running mean over all earlier calls on that engine, warm-ups included. This PR resets the tracker per call and
recomputes the published acceptance figures from the per-step counts the result files already store. No GPU run is
needed.

How it was found

A step-by-step trace recorded on a T4 for a web replay (flashspec_trace_t4.py, 0.3.1, 2 warm-ups and 3 repeats per
prompt) reported acceptance values that did not match the trace's own steps. Replaying the trace's call order through
a running tracker reproduces all 20 reported values exactly. For example, HumanEval/1:rename reported 0.3075, but its
own steps give 0.8790.

What is affected

  • Affected: GenerationResult.acceptance_rate on every call after the first; the harness's acceptance_rate_mean
    (a mean of those values); the README acceptance column; Figure 9 in docs/lessons.md.
  • Not affected: throughput and every speed-up, outputs and the losslessness tests, the bandit (update() gets each
    step's own counts), arm_steps and arm_report.

Across the 100 result files that store both values, acceptance_rate_mean differs from accepted / proposed draft
tokens (from arm_report) by a median of 0.0126 and at most 0.0913. The stored value is higher in 40 files and
lower in 60:

import json, statistics
from pathlib import Path
from benchmarks.arm_report import pooled_acceptance
d = []
for f in sorted(Path("benchmarks/results").rglob("*.json")):
    doc = json.loads(f.read_text())
    if not isinstance(doc, dict): continue
    stored = (doc.get("metrics") or {}).get("acceptance_rate_mean")
    counted = pooled_acceptance(doc.get("arm_report"))
    if stored is not None and counted is not None:
        d.append(counted - stored)
print(len(d), statistics.median(map(abs, d)), max(map(abs, d)), sum(x < 0 for x in d), sum(x > 0 for x in d))

The 0.2.0 run (8d036d7) and older files have no arm_report, so they cannot be recomputed. docs/benchmarks.md
says so next to its historical tables.

Changes

  1. fix(engine): self._acceptance_tracker.reset() at the start of both generate paths (batch-1 incremental and
    the general step loop), and a sharper acceptance_rate docstring. New tests/unit/test_acceptance_per_call.py:
    several calls on one engine at batch 1 and 2, greedy and sampled. Each rate must equal that call's own
    accepted / proposed from arm_steps.
  2. fix(readme): benchmarks.arm_report.pooled_acceptance(report) returns accepted / proposed summed over arms.
    scripts/ingest_gpu_bundle.py uses it for the acceptance cell in both tables, or n/a without arm_report. The
    table notes say this cell is computed, not stored. README regenerated with --readme-commit 3c43e08.
  3. docs(lessons): Figure 9 uses pooled_acceptance. The caption changes from 0.94 to 0.95 and from 0.45 to 0.54;
    161 tok/s and 70 tok/s are unchanged. The qwen-g2 label moved inside the plot, and the subtitle, which was cut off
    at the right edge, is shorter. Lesson 7 adds this bug: no test had made two calls in a row on one engine.
  4. CHANGELOG [Unreleased] → Fixed.

Before / after: README acceptance (bundle 3c43e08, every other cell unchanged)

table dataset mode method before (acceptance_rate_mean) after (arm_report counts)
status humaneval greedy flashspec_compiled 0.3919555827630965 0.3775609756097561
status humaneval T=0.7 flashspec_compiled 0.3852643082609558 0.3888888888888889
status mt_bench greedy flashspec_compiled 0.2868673245081164 0.2789115646258503
status mt_bench T=0.7 flashspec_compiled 0.30095750082855754 0.2970561998215879
full humaneval greedy flashspec_ucb 0.3909112890710105 0.3759727626459144
full humaneval T=0.7 flashspec_ucb 0.3852359399073896 0.3885480572597137
full mt_bench greedy flashspec_ucb 0.2620603532680512 0.2712363330529857
full mt_bench T=0.7 flashspec_ucb 0.2884380538950996 0.2892670157068063

The full table's flashspec_compiled rows show the same before and after values as the status table.

Gate

Local, on this branch:

  • make lint (ruff, format, import-linter: 2 contracts kept), mypy --strict flashspec: clean
  • pytest tests/unit tests/integration -m "not gpu": 574 passed, 5 skipped (coverage 88.94%), including
    test_output_distribution.py
  • pytest tests/chaos: 5 passed
  • the new test fails on main (3/3) and passes here
  • python scripts/readme_api.py --check: current
  • scripts/article/make_030_figures.py: regenerated figures match (test_article_figures.py)
  • no change to Algorithm 1, the residual, or the sampling path

SpeculativeEngine created its AcceptanceTracker once and never reset it, so
GenerationResult.acceptance_rate was a running mean over every earlier
generate() call on the engine, warm-ups included, although the field is
documented as the rate for this call. The tracker now resets at the start of
each call, on both the batch-1 incremental path and the general step loop.

Throughput, outputs, the bandit's updates and arm_steps were not affected.
The new test makes several calls on one engine (batch 1 and 2, greedy and
sampled) and checks each rate against that call's own arm_steps counts; it
fails on main (3/3) and passes here.
acceptance_rate_mean in result files from 0.3.1 and earlier averages the
engine's running means, so the README tables now show accepted / proposed
draft tokens over the timed calls, computed from the stored arm_report counts
by the new benchmarks.arm_report.pooled_acceptance (n/a without them). The
table notes say this cell is computed, not stored.

README regenerated with --readme-commit 3c43e08: only the acceptance cells
and the two notes change. The unpacked result files are byte-identical.
…as a lesson

Figure 9 plots accepted / proposed from each arm's stored arm_report counts
(Qwen g2 0.95, lookup g6 0.54; throughput unchanged), and the caption follows.
The qwen-g2 label moves inside the plot and the subtitle no longer runs past
the right edge. Lesson 7 gains the acceptance tracker as another path no test
had run (two calls on one engine). docs/benchmarks.md notes that its older
acceptance values cannot be recomputed.
@Mattral
Mattral merged commit e974e1e into main Oct 29, 2026
6 checks passed
@Mattral
Mattral deleted the fix/acceptance-per-call branch October 29, 2026 03:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant