What Is Prefill and Decode in AI Inference?

Prefill and decode are the two phases every language model request passes through. Prefill reads your entire prompt at once. Decode writes the answer one token at a time. They run on the same hardware, but they stress completely different parts of it, and almost every strange thing about LLM pricing and latency falls out of that difference.

If you have ever wondered why output tokens cost more than input tokens, why the first word takes ages and the rest arrive quickly, or why prompt caching saves so much more than it seems like it should, this is the mechanism underneath all three.

Prefill: reading the prompt

When you send a request, the model has to build an internal representation of every token in your prompt before it can generate anything. This is prefill.

The important property is that prefill is parallel. Every token in your prompt can be processed at the same time, because they are all already known. A 10,000-token prompt does not require 10,000 sequential steps. It requires one pass over a large batch of tokens, which is exactly the shape of work GPUs are built for: big, dense matrix multiplications with high arithmetic intensity.

Prefill is therefore compute-bound. You are limited by how many floating-point operations the chip can do per second. The hardware runs close to its theoretical peak, and the work scales with prompt length.

Prefill is also what you are waiting for during time to first token. A long prompt means a long prefill means a long pause before anything appears.

Decode: writing the answer

Generation is different in kind, not just degree. The model produces one token, appends it to the sequence, and only then can it produce the next. Token 40 cannot be computed before token 39 exists. Decode is inherently sequential.

This has an unfortunate hardware consequence. Each decode step processes a single new token, but to do so it must read the model's entire set of weights out of memory. For a large model that is a lot of data moved to do a small amount of arithmetic.

Decode is therefore memory-bandwidth-bound. The chip's compute units sit substantially idle, waiting on memory. Adding more raw FLOPS barely helps. NVIDIA's own walkthrough of inference optimisation covers the same split in more depth. This is why inference hardware discussions obsess over memory bandwidth rather than peak compute, and why the same GPU that tears through prefill produces tokens at a pace that feels almost leisurely.

Why this explains the price asymmetry

Now the pricing makes sense. Across essentially every provider, output tokens cost meaningfully more than input tokens, often by a multiple rather than a margin.

That is not a margin decision. It reflects genuine cost:

• An input token is processed in parallel with thousands of others, at high hardware efficiency, in a single shared pass.

• An output token requires its own sequential step, each one dragging the full weight set through memory, at low hardware efficiency.

One input token and one output token are the same object in your billing line and completely different amounts of work in the datacenter. Why bigger models cost more per token is the same story from the model-size direction: more weights means more bytes moved per decode step.

Why the KV cache exists

There is an obvious inefficiency lurking in decode. To generate token 40, the model needs its representation of tokens 1 through 39. Recomputing all of them at every step would make generation quadratic and hopeless.

So it does not. The intermediate key and value tensors from prefill are kept in memory and reused at every decode step. That is the KV cache, and it is the single reason generation is tractable at all.

It also explains a cost that catches people out. The KV cache grows with sequence length and lives in expensive GPU memory for the whole request. Long contexts are not just more prefill work up front, they are a larger memory footprint held for the entire generation, which reduces how many requests the provider can run concurrently on one chip. That is a real part of why long context costs more even when you are generating the same short answer.

Why prompt caching saves so much

Prompt caching stores the prefill result for a prompt prefix so a later request with the same prefix can skip recomputing it.

Given the phase split, the size of that saving is predictable. Caching eliminates prefill work, which is the compute-heavy, parallel, prompt-length-scaled part. It does nothing for decode. So:

• A request with a 20,000-token system prompt and a 50-token answer is almost entirely prefill. Caching is close to transformative.

• A request with a 200-token prompt and a 2,000-token answer is almost entirely decode. Caching saves you nearly nothing.

If prompt caching has underdelivered for you, check that ratio. The technique is not underperforming, it is being applied to a workload whose cost lives in the other phase.

What to do with this

Three practical rules follow directly.

Optimise the phase that dominates your workload. Measure your average input and output token counts. Long-in, short-out workloads such as summarisation, extraction and classification are prefill-dominated: attack them with caching and prompt trimming. Short-in, long-out workloads such as drafting and code generation are decode-dominated: attack them by generating less, or by using a smaller model, since decode cost tracks model size closely.

Do not expect batching to help uniformly. Providers batch requests to keep hardware busy. Batching helps decode considerably, because it amortises the weight-loading cost across several sequences. It helps prefill much less, because prefill was already saturating the chip.

Read latency as two numbers, not one. Time to first token is your prefill. Tokens per second afterwards is your decode. A request that feels slow because of a 6,000-token system prompt needs a different fix than one that feels slow because it is writing 3,000 tokens of output. Collapsing both into "latency" hides which lever to pull. The cost side of the same reasoning is in how to reduce AI API costs, and the wider mechanics live in how AI models work.

FAQ

Is prefill the same as prompt processing? Yes. Prefill, prompt processing and the prompt phase all refer to the same step: building the model's internal representation of your input before generation starts.

Why is the first token slow but the rest fast? The first token waits for prefill over your whole prompt. Subsequent tokens each need only one decode step reusing the KV cache. Longer prompts lengthen the first wait without slowing the tokens after it.

Does decode get faster with a better GPU? Only if the better GPU has more memory bandwidth. Decode is bandwidth-bound, so extra compute capacity largely goes unused. This is why bandwidth is the headline number for inference hardware.

Does this apply to models running locally? Yes, and more visibly. On consumer hardware the bandwidth ceiling is lower, so the gap between fast prefill and slow decode is wider. See running AI models locally.