Speculative Decoding: Cutting LLM Inference Latency Without Touching Model Quality

If you're serving an LLM in production, you've hit this wall: the model is accurate enough, but it's too slow, and every trick to speed it up — quantization, batching, smaller models — trades away quality to get there. Speculative decoding is the rare exception. It gets you 2-3x more tokens per second with mathematically identical output to the original model. No quality tradeoff, no retraining required in the simplest version. That's why it's already running under the hood at Google, Anthropic, and every major inference provider.

Most engineers know it exists but have never turned it on, because the explanations are either hand-wavy ("use a small model to speed up a big model") or buried in research papers about tree attention and acceptance rates. This post explains the mechanism precisely enough that you can reason about when it will and won't help, then shows you how to enable it in vLLM in about ten lines of config.

The problem: autoregressive decoding is serial by construction

A transformer generates text one token at a time. To produce token N, it needs token N-1 already committed, because that token gets fed back in as part of the input. This is why LLM inference is latency-bound rather than throughput-bound for a single request — you can't parallelize across tokens within one sequence, no matter how much GPU you throw at it.

Here's the part that matters: a forward pass over a large model, given a full sequence, is not much slower than a forward pass over one appended token. GPUs are massively parallel, and processing 5 tokens in one pass costs barely more than processing 1, because the bottleneck is loading model weights from memory (memory-bandwidth bound), not the matrix multiplies themselves. This is the exact inefficiency speculative decoding exploits: if you're going to pay the cost of a forward pass anyway, you may as well verify several candidate tokens in it instead of just one.

How it actually works

Speculative decoding runs two models together:

1. A draft model (small, fast, cheap) proposes the next k tokens autoregressively — say k=4 or k=5. This costs almost nothing because the draft model is 10-100x smaller than the target model.
2. The target model (your real, large model) runs a single forward pass over the entire draft sequence at once — verifying all k proposed tokens in parallel, the same way it would score any input sequence.
3. For each position, the target model's output distribution is compared against what the draft model predicted. Tokens are accepted left-to-right until the first mismatch (using a rejection-sampling rule that guarantees the final output distribution is identical to sampling from the target model alone — this is the part that makes it "free" quality-wise, not an approximation).
4. Every accepted token is a full step of decoding, done for the price of the draft model's cost. The first rejected token gets resampled correctly from the target model's own distribution, so you always make forward progress.

The upshot: if the draft model correctly predicts 3 of 4 tokens on average, you get roughly 4 tokens of output for the cost of 1 target-model forward pass plus 4 cheap draft passes. That's the 2-4x speedup you see reported.

Draft model:  "The"  "capital"  "of"  "France"  →  proposes 4 tokens
Target model: verifies all 4 in ONE forward pass
Result:       "The" ✓  "capital" ✓  "of" ✓  "France" ✓  → all accepted, next round starts
                                              (if "France" had been wrong, target
                                               resamples correctly at that position)

Why the output distribution doesn't change

This is the detail that convinces skeptical engineers it's not "just an approximation." The acceptance rule is a form of rejection sampling: a draft token is accepted with probability min(1, p_target(token) / p_draft(token)). If rejected, the next token is sampled from a corrected residual distribution max(0, p_target - p_draft), renormalized. Mathematically, sampling through this two-stage process is provably equivalent to sampling directly from the target model. You are not swapping in the small model's judgment anywhere — you're only using it to generate candidates that get checked.

Choosing a drafting strategy

You don't have to train a custom draft model to get started. There are four common approaches, in increasing order of setup effort and typical speedup:

Approach	How it works	Setup effort	Typical speedup	When to use
Off-the-shelf small model	Use an existing smaller model from the same family (e.g., Llama-3.2-1B drafting for Llama-3.1-70B)	Low — just point to two checkpoints	1.5-2x	Quick win, same-family models available
Prompt lookup / n-gram matching	No model at all — draft tokens by matching n-grams already in the prompt/context (great for code, retrieval-heavy tasks)	Very low — no extra model to host	1.5-3x on repetitive text	RAG, code completion, editing tasks with high context overlap
EAGLE / Medusa (auxiliary heads)	Add small prediction heads on top of the target model's own hidden states — the "draft model" is really a few extra layers	Medium — requires the head weights (often released alongside the base model)	2-3x	You want speed without hosting a second full model
Distilled draft model	Train a small model specifically distilled from the target model's outputs	High — needs a training run	2.5-4x	High-volume production serving where the investment pays back quickly

Start with prompt lookup decoding or an off-the-shelf small model from the same family — both give a meaningful win with near-zero setup, and you can graduate to EAGLE-style heads once you've confirmed the workload benefits.

Turning it on in vLLM

vLLM has first-class support for speculative decoding. Here's a minimal setup using an off-the-shelf draft model:

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    speculative_config={
        "model": "meta-llama/Llama-3.2-1B-Instruct",
        "num_speculative_tokens": 5,
    },
    tensor_parallel_size=4,
)

params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(["Explain the CAP theorem in two sentences."], params)
print(outputs[0].outputs[0].text)

If you don't want to host a second model at all, use n-gram-based prompt lookup decoding instead — useful when your workload has heavy overlap between the prompt and the expected output (RAG answers that quote the context, code edits, summarization):

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    speculative_config={
        "method": "ngram",
        "num_speculative_tokens": 5,
        "prompt_lookup_max": 4,
        "prompt_lookup_min": 2,
    },
)

Check num_speculative_tokens first if a change in setup doesn't produce the expected speedup — pushing it too high (8+) can hurt throughput, because every rejected token wastes the draft compute spent on tokens after it. 3-5 is a reasonable starting range for most workloads.

Measuring whether it's actually helping

Speculative decoding's payoff depends entirely on the acceptance rate — how often the draft model's guesses match what the target model would have produced. This varies hugely by workload:

• High acceptance (60-90%): factual completions, code with a fixed template, structured output (JSON schemas), anything where the "obvious next token" is genuinely obvious
• Low acceptance (20-40%): creative writing, open-ended reasoning, high-temperature sampling — the target model's distribution is genuinely uncertain, so a small model can't predict it well

If your vLLM deployment exposes speculative decoding metrics, watch spec_decode_num_accepted_tokens vs spec_decode_num_draft_tokens. An acceptance rate under ~35% usually means the draft model isn't a good match for this workload — either it's too different from the target model, or the task is inherently high-entropy and speculative decoding won't help much no matter which draft model you pick.

Common pitfalls

• Assuming it always speeds things up. At low temperature and low acceptance rate, the draft model's forward passes are pure overhead. Benchmark on your actual traffic, not a demo prompt.
• Ignoring memory cost. You're now hosting two models. On a memory-constrained deployment, the draft model's KV cache and weights compete with the target model for VRAM — this can shrink your effective batch size and offset the latency win at scale.
• Confusing this with model distillation for accuracy. Speculative decoding uses the draft model only as a proposal generator whose guesses get checked and corrected. It has zero effect on final output quality — don't reach for it as a shortcut for actually shrinking your serving model.
• Picking a draft model that's architecturally mismatched. Vocabulary and tokenizer must match the target model exactly, or the token-level verification breaks down. Same-family models (Llama drafting for Llama, Qwen drafting for Qwen) are the safe default.
• Over-tuning num_speculative_tokens on one prompt. Acceptance rate varies by task type within your own traffic. Tune against a representative sample of production requests, not a single benchmark prompt.

Next steps

If you're serving models yourself (vLLM, TGI, SGLang, or a custom stack), speculative decoding is one of the highest-leverage latency optimizations you can ship this week — most serving frameworks now support it as a config flag, not a rewrite. If you're consuming a hosted API instead, know that this is often already running behind the scenes; ask your provider whether their fast/turbo tiers use speculative decoding, since it explains a lot of the latency gap between "standard" and "fast" model variants at the same price.

Want to go deeper on production LLM serving, inference optimization, and the systems-level tradeoffs behind building AI products that actually perform at scale? That's exactly what we cover hands-on in the Joinloop AI Engineering cohort.