What Is a Cold Start in AI Inference?

A cold start in AI inference is the extra delay on the first request, before the system is in a state to answer at its normal speed. The reason it confuses people is that it is not one delay. Three different things hide behind the phrase, they have different magnitudes, and the standard fix of keeping something warm only addresses one of them. Knowing which one you are hitting is most of the work, because the remedies have nothing in common.

The three delays, in the order they happen

Only the first is a cold start in the strict, borrowed-from-serverless sense. The other two get the same name in practice because they look identical from the outside: you send a request, and it takes much longer than the one after it.

Stage 1: the runtime has to exist

On any platform that scales to zero, the first request after an idle period pays for a process that has to be scheduled, started and made ready. On GPU infrastructure this is worse than on ordinary serverless, because the scheduler may also be waiting for an accelerator to free up.

This is the delay that disappears if you pay to keep an instance running. It is also the only one of the three where the fix is purely commercial, and it is why provisioned capacity costs what it does.

Stage 2: the weights have to be in memory

A model cannot answer until its weights are loaded into GPU memory, and that read is proportional to the file. A checkpoint of a few gigabytes loads quickly. An 80 GB checkpoint does not, regardless of how fast the model generates tokens afterwards. This is the stage people underestimate, because generation speed and load time are unrelated numbers that both get called performance.

Two things shrink it: a smaller or quantised model, and storage close to the compute. Our explainer on how much VRAM a local model needs covers the memory arithmetic that governs this, and it is the same arithmetic whether the GPU is in your machine or someone else's data centre.

If you call a large hosted API, you almost never see this stage. The provider keeps popular models resident precisely so you do not. It becomes your problem the moment you self-host or pick a model obscure enough that nobody keeps it loaded.

Stage 3: the cache has not seen this prompt before

The third delay is not about infrastructure at all. Before a model generates anything it must process your prompt, and with a long system prompt or a large attached document that processing is real work. Providers cache the result for repeated prefixes, so the second identical opening is far cheaper than the first. A miss puts the full cost back.

This is why the same application can feel fast all afternoon and slow after you edit one line near the top of the system prompt: you invalidated the prefix everything else was built on. Our explainer on prompt caching covers how to structure prompts so the stable part stays cacheable, and the KV cache explains what is actually being kept.

Telling them apart

You can usually identify the stage without any instrumentation:

• Slow only after a long idle period, then fast for everything that follows: runtime start.

• Slow on the first request to a particular model, including right after a deployment, and proportional to model size: weight loading.

• Slow whenever the beginning of the prompt changes, even on a busy system: prompt cache miss.

• Slow on every request regardless: not a cold start at all. That is steady-state latency, and our comparison of latency and throughput is the right place to look.

Measuring the gap between sending a request and receiving the first token separates the first two from the third reasonably well, since both infrastructure stages delay everything while a cache miss mostly delays the start of generation. Our explainer on time to first token covers that measurement.

What to do about it

In rough order of how much they return for the effort: keep the prefix of your prompts stable so caching works, pick the smallest model that passes your evaluations rather than the largest you can afford, and only then pay for provisioned capacity. Reserving a warm instance is the most expensive fix and solves only stage one, which is why it disappoints people who were actually hitting stage three.

One pattern worth avoiding: a synthetic request on a timer to keep things warm. It works against runtime start, does nothing for a cache miss on a different prompt, and bills you for every ping. Where request volume is genuinely sporadic, scheduled batch processing often fits the problem better than trying to keep an interactive path warm. For the wider picture of what happens between your request and the first token, see our guide to how AI models work.

Frequently asked questions

Why is the first AI request always slower?

Because one or more of three things has to happen first: a runtime is started, model weights are read into GPU memory, or your prompt prefix is processed without a cache hit. Later requests skip whichever of those is already done.

Do cold starts affect hosted APIs like the big providers?

Rarely in the weight-loading sense, because popular models are kept resident. Prompt cache misses still affect you, and they are the version of the problem most API users actually experience.

Does a smaller model remove cold starts?

It shortens weight loading substantially, since that stage scales with file size. It does not remove runtime start or cache misses.

Is keeping a model warm worth the cost?

Only if your delay is genuinely runtime start and the latency matters to users. If the slow path is a prompt cache miss, reserved capacity costs money and changes nothing.