06 — Speculative Decoding: The Full Lineage

A family of techniques for making decode faster by spending the idle compute (decode is memory-bound; tensor cores sit idle) on speculative tokens that get verified in parallel. Mathematically lossless when done correctly.

The core idea in three sentences

  1. Decode is memory-bound; a forward pass on the target model can actually process many tokens at once for roughly the same wall-clock time as one (as long as they fit in the compute budget).

  2. If a cheap drafter proposes k candidate tokens, the target model can verify all k in one forward pass by feeding them as if they were already generated and comparing distributions.

  3. Accept the longest prefix of drafter tokens that pass a rejection sampling test; sample the next real token from a corrected distribution; discard the rest. Repeat.

Expected speedup ≈ average accepted tokens per iteration. Practical numbers: 2-4× for a good drafter on chat workloads.

The math (why it’s lossless)

Drafter proposes token x_i with probability q(x_i). Target model assigns probability p(x_i). Rejection sampling rule:

  • If p(x_i) >= q(x_i): accept always.

  • If p(x_i) < q(x_i): accept with probability p(x_i) / q(x_i). On rejection, resample from the residual distribution max(0, p(x) - q(x)) renormalized.

This guarantees the output distribution is exactly p — identical to sampling from the target model alone. The final token distribution is provably the same as autoregressive sampling. No quality tradeoff, only speed.

(For greedy decoding the rule collapses to “accept if the drafter’s top-1 matches the target’s top-1” — also lossless.)

Read Leviathan Appendix A once. It’s five lines.

The lineage, in publication order

1. Original speculative decoding — Leviathan et al. (Google), Chen et al. (DeepMind), both 2023

  • Leviathan et al. — “Fast Inference from Transformers via Speculative Decoding” (ICML 2023). arXiv: <phone_number_or_numberic_id_or_random_id_33>.

  • Chen et al. — “Accelerating Large Language Model Decoding with Speculative Sampling”. arXiv: <phone_number_or_numberic_id_or_random_id_34>. Published within days; same core idea; slightly different framing.

Uses a separate small draft model (e.g., use Llama-1B to draft for Llama-70B). Reported: 2-2.5× speedup on Chinchilla-70B decoding.

Drawbacks that motivated everything after:

  • Need to train or select a compatible small model (same tokenizer, compatible distribution).

  • Draft model takes GPU memory + compute, which competes with the target model’s batch.

  • Two model families = two deployment complications.

2. Medusa — Cai et al., 2024

arXiv: <phone_number_or_numberic_id_or_random_id_35>. https://github.com/FasterDecoding/Medusa

Insight: don’t use a separate model. Attach k additional decoding heads to the target model, each predicting a token at offset +i. Use tree attention to verify multiple candidate paths in one pass.

Two variants:

  • Medusa-1: frozen backbone, only heads are trained. Cheap to add to any pretrained model.

  • Medusa-2: joint fine-tuning of backbone + heads. Higher quality, more training cost.

Reported: 2-3× speedup with acceptance rates around 2-2.5 tokens per step. Weakness: the k independent heads at offsets +1, +2, +3 don’t condition on each other, so their joint distribution is worse than a sequential drafter.

3. EAGLE-1, EAGLE-2, EAGLE-3 — Yuhui Li et al., 2024-2025

Current SOTA for speculative decoding. All from the SafeAILab group. Repo: https://github.com/SafeAILab/EAGLE.

  • EAGLE-1 (ICML 2024): instead of predicting tokens, predict hidden features autoregressively at a lower-dimensional level. A small transformer drafter is trained to predict the target’s next hidden state given past hidden states + past tokens. Output is projected back to token distribution. Feature-level autoregression is easier than token-level (features are smoother), so acceptance rate climbs to ~3-4 tokens/step.

  • EAGLE-2 (EMNLP 2024): adds a context-aware dynamic draft tree. The drafter emits a tree of candidates whose width and depth adapt per token — be aggressive when the drafter is confident, be conservative when it’s not. Big empirical win from the same training.

  • EAGLE-3 (NeurIPS 2025). arXiv: <phone_number_or_numberic_id_or_random_id_36>. The current state of the art. Two changes:

    1. Abandons feature prediction (turns out the feature-level autoregression scales poorly with more training data — the drafter starts overfitting to the target’s features rather than learning generalization).

    2. Introduces “training-time test” + multi-layer feature fusion: the drafter is trained to predict tokens directly, using concatenated features from multiple target-model layers as input, with a curriculum that mimics inference-time acceptance patterns.

    Results: up to 6.5× speedup, ~1.4× over EAGLE-2. Acceptance length 4-6 tokens per step on MT-bench with Llama 3.1 8B.

vLLM has production-grade EAGLE-2 and EAGLE-3 support in vllm/v1/spec_decode/. SGLang’s EAGLE support is experimental as of mid-2026.

4. MTP — Multi-Token Prediction (DeepSeek-V3 style)

Instead of adding speculative heads at inference, bake multi-token prediction into training. The model has extra prediction modules for tokens at offset +1, +2 (typically just +1 for DeepSeek-V3), trained jointly with the main next-token loss.

At inference time these extra heads become natural drafters — same weights, same distribution, no separate drafter model. DeepSeek-V3 reports meaningful speedup with MTP-1 (one extra token) while also improving base model quality (multi-token loss acts as a regularizer).

SGLang has explicit support for DeepSeek MTP as a speculative decoding backend. This is one place SGLang leads.

5. n-gram / prompt-lookup decoding — Apoorv Umang, 2023

GitHub: https://github.com/apoorvumang/prompt-lookup-decoding. Integrated into both vLLM (prompt_lookup_num_tokens=10 on the request) and HF transformers.

The nearly-free trick. No drafter model at all. For each decode step:

  1. Take the last N tokens of the current generated sequence.

  2. Search for that N-gram in the prompt.

  3. If found, propose the next k tokens from the prompt as the draft.

  4. Verify in one forward pass. Accept whatever matches.

Zero training. Zero extra parameters. Works spectacularly on any task where the output copies from or paraphrases the input:

  • Code editing / refactoring (huge win — model regurgitates most of input file)

  • RAG / document QA (model copies quotes from context)

  • Translation / summarization with lots of copy

Fails silently on divergent generation (creative writing, open-ended chat) — acceptance rate drops to zero, only cost is the extra n-gram lookup (negligible).

For your Zoho agentic use case: enable this for tool-call output generation. Tool call arguments often echo the user query or context; n-gram lookup will grab them nearly for free.

When does speculation actually pay?

Crucial economic point often missed: speculation trades compute for bandwidth. It only wins when compute was idle.

Scenario

Does spec decoding pay?

Batch 1, decode-only, memory-bound

YES, biggest wins here. Tensor cores idle; use them to verify k tokens.

Small batch (2-4), decode-only

Yes, still memory-bound at 8B scale

Medium batch (16-32)

Marginal. Compute is starting to matter.

Large batch (64+)

NO. Compute-bound now; verifying k tokens actually slows down the batch because the forward pass takes longer.

Prefill

No. Already compute-bound.

Production lesson: speculative decoding is a latency feature, not a throughput feature. Enable it on latency-sensitive tiers (small batches, chat), disable it on high-throughput batch pipelines.

vLLM V1’s scheduler is aware of this and can be configured to skip speculation above a batch-size threshold. Read vllm/v1/spec_decode/ for the current logic.

Acceptance rate reality check (mid-2026)

Rough current numbers on Llama-3.1-8B-Instruct, MT-bench:

Method

Avg accepted tokens / step

Effective speedup

N-gram lookup (RAG-ish task)

3-8

2-4×

Medusa-2

2.0-2.5

1.8×

EAGLE-1

3.5-4.0

2.5-3×

EAGLE-2

4.0-5.0

3-4×

EAGLE-3

4.5-6.0

4-6.5×

DeepSeek MTP-1

~2

1.7-1.9×

These are ideal-case numbers on benchmarks; real workloads run ~70-80% of these.

The headline: EAGLE-3 + small batch = 4-6× speedup at zero quality cost. This is the biggest “free” latency win in the field right now.

The vLLM configuration surface

# Speculative config in vLLM V1 (JSON or YAML in --speculative-config)
{
    "method": "eagle3",             # or "eagle", "eagle2", "medusa", "ngram", "draft_model"
    "model": "path/to/eagle3-head", # required for eagle*/medusa
    "num_speculative_tokens": 5,     # draft tree depth
    "draft_tensor_parallel_size": 1, # usually 1 for eagle
}

For n-gram, per-request:

{ "prompt_lookup_num_tokens": 10 }

(No speculative-config needed for n-gram.)

Reading exercise

  1. Read Leviathan Appendix A. Reproduce the rejection sampling proof on paper.

  2. Read EAGLE-3 §3. Answer: what is the “training-time test”, and why does it change the failure mode of EAGLE-1?

  3. Enable n-gram lookup on your local llama.cpp or vLLM setup with a RAG-style prompt. Measure the acceptance rate empirically. It should be > 3.

The intuition to internalize

Speculation is monetizing idle tensor cores. Every decode step your GPU could execute a much bigger forward pass than it does; spec decoding trades that spare compute for a proposal-verify cycle. The lineage (Leviathan → Medusa → EAGLE → EAGLE-3 → MTP → n-gram) is a series of progressively cheaper ways to generate the proposal. The math (rejection sampling) guarantees zero quality cost. The economics (batch-size sensitivity) tell you when to turn it on.

On your Zoho agent tier: enable it. On the batch-analytics tier: don’t. When a customer says “why is streaming slower under high load?” you now know one of the answers is “spec decoding disabled itself because the batch grew past the threshold, which was correct behavior.”