Speculative Decoding: How a Small Draft Model Speeds Up LLM Generation Without Changing Output
Break the memory-bandwidth bottleneck in autoregressive decoding: draft-and-verify mechanics, acceptance lengths, and verification math.
On this page
When running inference on large language models, users frequently ask why generating tokens takes so much longer than processing the initial prompt.
A user submits a 2,000-token prompt: the model computes the prefill in 200 milliseconds (processing at 10,000 tokens per second). But when generating the answer, the speed drops to 35 tokens per second.
This dramatic slowdown is caused by the memory-bandwidth bottleneck of autoregressive decoding.
In standard autoregressive generation, generating each single token requires loading all 70 billion parameters (140 gigabytes of weights) from High-Bandwidth Memory (HBM) into the GPU’s arithmetic processing cores. The GPU’s Tensor Cores spend 95% of their time sitting completely idle, waiting for weights to travel over the memory bus.
Speculative Decoding solves this fundamental hardware limitation, speeding up token generation by 2x to 3x while mathematically guaranteeing that the output distribution is 100% identical to the target model.
+--------------------------------------------------------------------------+
| THE SPECULATIVE DECODING WORKFLOW |
| |
| [Small Draft Model (1B Params)] ---> Speculatively proposes 5 tokens: |
| "The", "capital", "of", "France", "is"
| |
| [Large Target Model (70B Params)] -> Evaluates all 5 tokens in PARALLEL |
| in a single forward pass. |
| |
| Result: 4 tokens accepted + 1 correction emitted. |
| Outcome: 5 tokens generated in the time of 1 traditional forward pass! |
+--------------------------------------------------------------------------+
The Hardware Reality: Compute-Bound vs. Memory-Bound
To understand why speculative decoding works, examine the arithmetic intensity of a GPU:
- Prefill Phase (Prompt Processing): Highly parallel. The GPU multiplies a matrix of prompt tokens against the weight matrix. The arithmetic intensity is high, fully saturating GPU compute.
- Decode Phase (Token Generation): Sequential. The GPU multiplies a single vector (the latest token) against the weight matrix. The arithmetic intensity is abysmal: the GPU transfers 140 GB of memory across the bus just to perform a tiny number of calculations.
Crucially: Evaluating 5 tokens in parallel takes virtually the exact same time as evaluating 1 token. The memory transfer cost is identical; the Tensor Cores simply perform 5 vector-matrix dot products instead of 1.
The Draft-and-Verify Algorithm
Speculative decoding pairs a massive Target Model (M_target, e.g., 70B parameters) with a lightweight Draft Model (M_draft, e.g., 1B parameters).
The execution follows a strict three-step cycle:
Step 1: Draft Generation
Draft Model generates K tokens sequentially (e.g., K = 4):
[x_1, x_2, x_3, x_4]
Step 2: Parallel Target Verification
Target Model evaluates all 4 tokens simultaneously in a single forward pass:
Computes probability distributions P(x_1), P(x_2), P(x_3), P(x_4)
Step 3: Stochastic Acceptance / Rejection
For each token i = 1..4:
Sample random r ~ U(0, 1)
If r < P_target(x_i) / P_draft(x_i):
Accept token!
Else:
Reject token! Resample correct token from adjusted distribution.
Discard all subsequent draft tokens.
The Rejection Math (Preserving Exact Mathematical Equivalence)
Why doesn’t the small draft model degrade the quality of the answer?
If a draft token is rejected at position $k$, the target model does not ask the draft model to retry. The target model corrects the token on the spot by sampling from an adjusted probability distribution:
P'(x) = max(0, P_target(x) - P_draft(x))
Normalized across the vocabulary.
Because of this exact rejection sampling formula, the probability distribution of accepted tokens is provably identical to sampling directly from the massive target model alone. The output is mathematically indistinguishable from running the 70B model autoregressively.
The Four Sources of Draft Tokens
Where does the draft model come from?
| Draft Source | How It Works | Typical Acceptance Rate | Primary Tradeoff |
|---|---|---|---|
| Small Separate Model | E.g., Llama-3-8B drafting for Llama-3-70B | 60% to 75% | Requires VRAM for second model weights |
| Medusa Heads | Single base model with multiple extra prediction heads | 50% to 65% | Fast; zero extra model loading; heads train cheaply |
| Self-Speculative (Layer Skipping) | Target model executes only every 3rd layer during drafting | 45% to 60% | Zero extra weights; requires custom CUDA kernels |
| Prompt Lookup Decoding | Copies repeating phrases directly from prompt context | 70% to 85% (on RAG / Code) | Near-zero compute; fails on creative prose |
Prompt Lookup Decoding (Zero-Model Speculation)
In tasks like code refactoring, document summarization, or RAG, the model frequently outputs variable names, citations, and phrases that already exist in the input prompt.
- Prompt Lookup Decoding uses classical string matching (n-gram lookup) over the prompt text to find candidate continuations.
- It requires zero neural draft model parameters. If the candidate n-gram matches, the target model verifies it in parallel; if not, it rejects it without penalty.
When Speculative Decoding Fails to Help
Speculative decoding is not a magic speedup for every scenario. It fails under two specific operating conditions:
[Low Acceptance Rate (Creative Writing / High Temperature)]
-> Draft model guesses incorrectly on every turn.
-> Acceptance length drops to 0.5 tokens.
-> Result: Slower than standard decoding due to draft overhead!
[High Concurrency GPU Saturation (Batch Size > 64)]
-> GPU memory bus is already 100% saturated by concurrent requests.
-> The extra matrix width of 5 draft tokens causes compute thrashing.
-> Serving engines dynamically disable speculation under high load.
- High Creative Entropy: In open-ended creative writing where the probability distribution is flat and temperature is high ($T > 1.0$), the draft model’s predictions diverge from the target model. If the acceptance rate drops below 40%, the overhead of running the draft model makes total generation slower than vanilla decoding.
- High-Concurrency Saturation: If a GPU server is already running at batch size 64 under peak corporate traffic, the memory bus is already fully utilized. Speculative decoding excels at low-concurrency, low-latency workloads (batch sizes 1 to 8), making it the premier optimization for real-time interactive agents.
Speculative decoding is a triumphant example of algorithmic hardware alignment: using cheap arithmetic headroom to overcome physical memory transfer limits.
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.