What Is Speculative Decoding in AI?
A small model guesses the next few tokens, the big model checks them all at once. Why that is faster, why it does not change the output, and what actually determines the speedup.
Speculative decoding is a way of making a large language model produce tokens faster without changing what it produces. A small, cheap draft model guesses the next several tokens, and the large model checks all of those guesses in a single pass instead of generating them one at a time. Guesses it agrees with are kept. The first one it disagrees with is replaced with its own choice, and the process repeats.
The part that surprises people: done correctly, the output is statistically identical to what the large model would have produced alone. You are not trading quality for speed. You are trading wasted hardware capacity for speed.
Why generating one token at a time is slow
To see why the trick works, you need one fact about how models run at inference time.
Generating a token requires reading the model's weights out of memory and pushing your input through them. For a large model, that read is enormous, and it dominates the cost. The actual arithmetic finishes long before the memory transfer does. This is what people mean when they call decoding memory-bandwidth-bound rather than compute-bound.
The consequence is counterintuitive: checking five tokens in one pass costs barely more than checking one, because you paid for the weight read either way. The compute units were sitting idle. Speculative decoding exists to fill that idle capacity with useful work.
The draft-and-verify loop
Mechanically, one round looks like this:
The draft model, typically far smaller and much faster, generates a short run of candidate tokens, often four to eight.
The target model, the big one whose output you actually want, processes all of those candidates in a single forward pass and produces its own probability for each position.
Candidates are accepted from the start of the run for as long as they are consistent with the target model's distribution. At the first rejection, the run stops.
The target model emits the correct token at the rejection point, so no round is ever wasted.
Repeat.
A useful way to hold it: the draft model is guessing what the big model was going to say, and the big model is marking the guesses. Because marking is parallel and generating is sequential, a good guesser turns five sequential steps into one.
Why it does not degrade quality
This is the part most explanations skip, and it is the reason the technique got adopted rather than remaining a curiosity.
The acceptance step is not "does this look close enough". It is a rejection-sampling procedure designed so that the tokens coming out the far end follow exactly the target model's probability distribution. When a draft token is rejected, the replacement is drawn from an adjusted distribution that corrects for the bias the draft introduced. The mathematics is set out in the research literature, including this survey of speculative decoding methods, and it is why implementations describe themselves as lossless.
So the draft model's quality does not affect what you get. It affects how often its guesses are accepted, which affects how much faster you get it. A poor draft model produces a correct answer slowly. A good one produces the same correct answer quickly.
What determines the speedup
Three things, and they interact.
Acceptance rate. The fraction of draft tokens the target model keeps. High on predictable text such as boilerplate code, formatting, or long quoted passages. Lower on genuinely uncertain content. This is why the same setup can feel dramatically faster on one workload than another.
Draft length. Propose too few tokens and you leave speed on the table. Propose too many and you spend draft-model time on tokens that get thrown away after the first rejection. Practical implementations tune this, sometimes dynamically.
Draft model cost. The draft has to be genuinely cheap. If it costs a third of the target model to run, it has to be right very often to be worth having. Inference-serving guides such as BentoML's write-up on speculative decoding go into how these are balanced in production systems.
Where the acceptance rate is low and the draft is expensive, speculative decoding can be slower than plain decoding. It is an optimisation, not a free win.
Why this matters if you are just using an API
You will rarely configure any of this yourself. It matters for one practical reason: when a provider ships a "fast" or "turbo" mode at a large speed multiple, speculative decoding is one of the common ways that number is achieved, and it means something different from the other ways.
If the speedup comes from a smaller model, the outputs change. You need to re-test your prompts.
If it comes from quantization, outputs can shift subtly. Worth checking. We cover the trade-off in what is quantization in AI.
If it comes from speculative decoding, the outputs do not change at all, and the only variable is latency.
That distinction is worth asking about before you re-run an evaluation suite you did not need to re-run. It also explains why speed improvements from this technique tend to be variable in a way that quantization is not: your acceptance rate depends on your content.
For the broader picture of what happens between your prompt and the response, see what is inference in AI and our overview of how AI models work.
FAQ
Does speculative decoding change the model's answers?
No, when implemented with the standard verification scheme. The procedure is designed to preserve the target model's output distribution exactly.
Is speculative decoding the same as using a smaller model?
No, and the difference is the whole point. A smaller model gives you different, usually worse output. Speculative decoding gives you the large model's output, sooner.
What is a draft model?
A small, fast model that proposes candidate tokens for the large model to check. It is often a smaller member of the same family, or a purpose-trained lightweight predictor.
Why is it faster on some prompts than others?
Because the speedup depends on how often the draft guesses correctly. Predictable, formulaic text has a high acceptance rate. Novel or highly uncertain text has a lower one.
Does it save money as well as time?
Not necessarily. It uses more total compute per token accepted, in exchange for lower latency. Whether that shows up as a lower bill depends entirely on how your provider prices it. Related savings usually come from elsewhere, such as prompt caching.
How did this land?
About the author
Senior Editor, AI & Product
Cecilia leads the Swarmz editorial desk. She has spent a decade turning complex AI and product topics into writing people actually finish, and she owns the blog's quality bar.
