## Speculative Decoding, Visualized

Large language models generate text one token at a time. Each token requires a full forward pass through billions of parameters. Speculative decoding makes this faster by using a small draft model to propose tokens, then verifying them all at once with the large target model.

### Overview

**Step 1 of 2**  
**Input Tokens**  
The sentence is tokenized into individual words.

## Why Generation Is Slow

You can't predict token 5 without first knowing token 4. The process is inherently sequential. A 70-billion parameter model that takes 50ms per token needs 5 full seconds to generate 100 tokens, waiting for each step to complete before starting the next.

But here's the asymmetry that makes speculative decoding possible: _verifying_ N tokens takes one forward pass, while _generating_ N tokens takes N forward passes. A language model can score an entire sequence in parallel, computing the probability distribution at every position simultaneously. If we could guess the next several tokens correctly, we could verify all of them in a single pass rather than generating them one by one.

## How It Works

The algorithm follows a draft-then-verify loop:

1. Generate K draft tokens using the draft model
2. Verify all K tokens in one target model forward pass
3. Accept tokens until a mismatch, then resample and discard the rest

In the best case, all K tokens are accepted, plus a bonus token sampled by the target model. That's K+1 tokens from a single pass. Even when some are rejected, you're guaranteed at least one token of progress.

### Draft Model

Let's take it step-by-step. We start by predicting six output tokens with our fast draft model.

## Preserving the Distribution

Speculative decoding produces samples from the exact target distribution, not an approximation. This requires careful handling of both acceptance and rejection.

### Compare Distributions

The draft and target models produce different probability distributions for each token. The animation above shows how this works. We compare target and draft probabilities at each token. The green region—the minimum of the two—represents guaranteed acceptance. The red excess above it is where the draft overestimated.

When we sample a token from the draft, what's the probability we accept it? It's simply the green height divided by the total draft height:

\[ p_{\text{accept}} = \frac{\min({\color{#b8864a}p_d}, {\color{#2a4a6d}p_t})}{{\color{#b8864a}p_d}} = \min(1, \frac{{\color{#2a4a6d}p_t}}{{\color{#b8864a}p_d}}) \]

If the target is higher than the draft, there's no red—we always accept. If the draft overestimated, we accept proportionally less.

When rejected, we can't simply resample from p_{target}—that would double-count tokens the draft already had a chance to propose. Instead, we sample from the residual distribution: the target minus the guaranteed acceptance, normalized to form a valid distribution.

\[ p_{\text{residual}}(x) = \frac{p_t(x) - \min(p_t(x), p_d(x))}{Z} = \frac{\max(0, p_t(x) - p_d(x))}{Z} \]

Accepted samples plus residual resamples combine to give exactly the target distribution. Same output, faster sampling.

### When Does It Help?

Speedup depends on acceptance rate and the cost ratio between models. The technique works best when:

- The draft aligns well with the target (distilled versions, same-family smaller models)
- The size ratio is large (70B/7B gains more than 7B/1B)
- Generation is predictable (code, structured output, common phrases)

Expect 2-3× speedup with a well-matched draft model. Worst case is worse than standard generation: you pay for both models but get just one token.

### Summary

Generating tokens is slow. Verifying them is fast. A small draft model guesses, a large target model checks in parallel. 2-3× speedup, identical output.