Skip to content

docs(advance): add Add a New Speculative Decoding Method guide#4639

Closed
Oxygen56 wants to merge 1 commit into
InternLM:mainfrom
Oxygen56:docs/issue-4589
Closed

docs(advance): add Add a New Speculative Decoding Method guide#4639
Oxygen56 wants to merge 1 commit into
InternLM:mainfrom
Oxygen56:docs/issue-4589

Conversation

@Oxygen56

Copy link
Copy Markdown

Motivation

The PyTorch engine has a clean plug-in surface for speculative decoding (BaseSpecProposer + SPEC_PROPOSERS registry in lmdeploy/pytorch/spec_decode/proposers/base.py), and four shipped methods register against it: eagle, eagle3, deepseek_mtp, qwen3_5_mtp. The user-facing docs/en/advance/spec_decoding.md teaches usage of those four names but never explains how to add a fifth, so users have asked the question externally.

Closes (partially) the docs side of #1738 and #4530.

Modification

Adds docs/en/advance/spec_decoding_new_method.md and a toctree entry for it in docs/en/index.rst, right next to spec_decoding.md.

The page covers:

  1. The registry / base class / method string triad.
  2. The build_specdecode_proposer entry point and why proposers/__init__.py must import the new class.
  3. What BaseSpecProposer already provides so contributors don't re-implement weight loading, draft forward, decoding-input update, or fallbacks.
  4. A minimal MyMethod(BaseSpecProposer) skeleton with @SPEC_PROPOSERS.register_module(name='my_method').
  5. The 3-tuple return contract for get_outputs (draft token ids, model_metas, target_hidden_states).
  6. When to override build_model, illustrated with the two in-tree precedents (Qwen3_5MTP shares the target embeddings; Eagle3 swaps embeddings conditionally and widens get_target_hidden_size).
  7. A 5-item shipping checklist.

No code changes. All snippets and references point to symbols that exist in lmdeploy/pytorch/spec_decode/proposers/.

BC-breaking

None — docs only.

Add a step-by-step documentation guide explaining how to add a new speculative decoding method to LMDeploy. Covers the SPEC_PROPOSERS registry system, BaseSpecProposer base class, the 3-tuple get_outputs contract, registration and import requirements, build_model override patterns, and a shipping checklist.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@RunningLeon

Copy link
Copy Markdown
Collaborator

It covers the same part as in #4589.

@lvhan028 lvhan028 closed this Jun 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants