Skip to content

docs: simpler quickstart - #55

Merged
Mattral merged 1 commit into
mainfrom
docs/simpler-quickstart
Oct 26, 2026
Merged

Mattral merged 1 commit into
mainfrom
docs/simpler-quickstart

Conversation

@Mattral

@Mattral Mattral commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

docs: simpler quickstart

Removes bandit=BanditConfig(n_arms=1) from the README quickstart and the docs examples wherever one model drafter is
used, and prints which drafter strategy ran. Docs and tests only; no library change. Do not merge until reviewed.

Why it can go

Under the default drafter_strategy="auto", SpeculativeEngine checks config.bandit.n_arms only when you pass a
BanditConfig yourself (check_n_arms = requested != "auto" or "bandit" in config.model_fields_set). With one model drafter,
"auto" resolves to composite-g6-f2, which is one arm, so the default BanditConfig() (n_arms=2) is never compared.

Changes

  • README quickstart:
    • drops the BanditConfig import and argument; the config is one line: FlashSpecConfig(device=..., sampling=SamplingConfig(temperature=0.0));
    • adds print("drafter strategy:", engine.drafter_strategy);
    • still 20 lines, ≤ 100 columns.
  • README recipes:
    • "Greedy or sampling" and "Turn compile off" drop bandit=BanditConfig(n_arms=1).
    • "Choose a drafter": the lookup-only example keeps it. It sets drafter_strategy="bandit" explicitly, and an explicit
      strategy still checks the arm count against the config. That recipe now imports BanditConfig itself.
  • docs/examples.md (the TinyLlama load_drafters example) drops BanditConfig(n_arms=1, strategy="ucb1") and says that
    "auto" runs composite-g6-f2 there.

Proof

New tests/unit/test_readme.py::test_quickstart_auto_runs_composite_and_matches_hf_greedy runs the README quickstart block as
written, on tiny models on CPU. It asserts:

  • there is no BanditConfig in the block;
  • engine.drafter_strategy == "composite-g6-f2", the reason is "auto: one model drafter…", and the arm is lookup 6 / fallback 2;
  • the printed output contains drafter strategy: composite-g6-f2;
  • all 128 generated tokens equal the target's generate(do_sample=False) (max_new_tokens=128, no EOS).

The existing README and examples code-block tests still run every block (examples included) with the bandit config removed.

Gate

Local, on this branch:

  • make lint (ruff, format, import-linter), mypy --strict flashspec: clean
  • pytest tests/unit tests/integration -m "not gpu": 559 passed, 5 skipped
  • python scripts/readme_api.py --check: current (no API change)
  • CI on this PR: docs-build pass, lint pass, onnx-parity pass, paper pass, test-chaos pass, test-cpu pass

Merging this before #54 or after makes no difference; they touch different files.

…nt the drafter strategy)

Under drafter_strategy="auto" the arm count follows the drafters when no
BanditConfig is passed, so the quickstart, two recipes and the examples page drop
bandit=BanditConfig(n_arms=1). The lookup-only recipe keeps it (explicit
"bandit" strategy). The quickstart prints engine.drafter_strategy. A new test
runs the quickstart block as written: composite-g6-f2, 128 tokens equal to HF
generate(do_sample=False).
@Mattral
Mattral merged commit b1c0ed6 into main Oct 26, 2026
6 checks passed
@Mattral
Mattral deleted the docs/simpler-quickstart branch October 26, 2026 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant