What Is a KV Cache? Why Long AI Chats Slow Down

A KV cache is the working memory a language model keeps while it writes a reply. As it processes your prompt, every token produces a pair of intermediate values, a key and a value, and the model stores them so it does not have to recompute them for the next token. Ask what is a KV cache in AI and the short answer is: it is the reason generating the second word is far cheaper than generating the first.

That sounds like an implementation detail. It is not, because three things you notice every day fall directly out of it.

The problem it solves

Transformers work by having every token attend to every token before it. To generate word number 500, the model needs a representation of all 499 that came before.

Without a cache, it would rebuild all 499 from scratch, then do it again for word 501, and again for 502. The cost of a reply would grow quadratically and long outputs would be unusable.

So the model keeps the keys and values from every token it has already processed. Generating the next token means computing one new key-value pair and attending over the stored set. As NVIDIA's write-up on inference optimisation puts it, this turns the per-token cost from a recomputation into a memory lookup.

The trade is straightforward. You buy speed with RAM.

Three things this explains

Why the first token is slow and the rest are fast

Sending a long prompt means the model must process all of it before it can write anything, filling the cache as it goes. That is the prefill phase, and it is compute-heavy.

Once the cache is warm, each subsequent token is a much smaller operation, which is the decode phase. That is why streaming output feels like a pause followed by a smooth flow rather than a steady trickle. You are watching prefill finish and decode start.

Paste in a 40-page document and the pause before the first word gets noticeably longer. The words after it do not.

Why long conversations get slower and more expensive

The cache grows with every token in the conversation, both yours and the model's. Each new message means more stored keys and values, more memory held, and more of them to attend over.

This is the mechanical reason a chat that has been running for two hours feels heavier than a fresh one, and why the cost per message climbs even when your messages stay the same length. You are paying to reprocess a conversation that keeps getting longer. The quality effects of that accumulation are a separate problem, covered in context rot.

Why context windows are a hard wall

A context window is often described as how much the model can pay attention to. It is more concrete than that. It is a budget for how large the KV cache is allowed to get.

The limit exists because the cache lives in GPU memory alongside the model weights themselves. Doubling the context roughly doubles the cache, and the hardware has a fixed amount of memory to divide between every request being served at once. That is why context limits move in discrete jumps tied to hardware and serving architecture rather than sliding upward smoothly.

KV cache is not prompt caching

These get conflated constantly, and the distinction is worth holding.

Prompt caching is built on the same idea, keeping computed keys and values around, but extends their life beyond one request so a repeated prefix does not need reprocessing. Providers document the mechanics and pricing themselves, for example Anthropic's prompt caching docs, and we covered the economics in what is prompt caching.

Short version: the KV cache always happens and you cannot turn it off. Prompt caching is a decision you make.

What to do with this

Put the stable part of your prompt first. System instructions, schemas, and reference documents that never change belong at the top, with the variable part at the bottom. Prompt caching works on shared prefixes, so a stable prefix is a cacheable prefix.

Start a new conversation rather than steering a long one. Once a thread has accumulated a few thousand tokens of history you are paying to carry all of it forward on every turn, usually for context you no longer need.

Read latency as two numbers. Time to first token tracks input length, time between tokens tracks output length. If your app feels slow, knowing which one is bad tells you whether to trim the prompt or shorten the response.

Do not read a bigger context window as a bigger brain. The cache holds everything but attention is not uniform across it, which is why a bigger context window does not automatically mean better answers.

FAQ

Does the KV cache mean the model remembers me?

No. It lives for the duration of a request, or for the cache lifetime you opted into, and then it is gone. Persistent memory across sessions is a separate feature built on storage and retrieval, described in how AI agents remember things between sessions.

Why can't providers just make context windows huge?

Because the cache sits in GPU memory that is also holding model weights and serving other users. Longer contexts mean fewer concurrent requests per GPU, which is a cost problem before it is a technical one.

Is a bigger KV cache better?

It is neither. It is a consequence of how much text is in play. What you want is the smallest cache that holds the context the task genuinely needs, since everything beyond that costs money and adds noise.

Do I need to think about this if I only use the API?

Mostly no, apart from two places it leaks through: prompt ordering, because prefix stability decides whether caching helps you, and conversation length, because it drives both cost and latency. Everything else is the provider's problem.