What Is Time to First Token? TTFT, Explained

Time to first token, usually shortened to TTFT, is the delay between sending a request to a language model and receiving the first piece of its answer. It covers queueing, reading your prompt and producing one token. It does not cover the rest of the response. A request with a 400 millisecond TTFT and one with a 4 second TTFT can finish at exactly the same moment, and users will describe the first as fast and the second as broken.

Time to first token, and the two numbers it gets confused with

Most latency complaints are really an argument between these:

They connect with arithmetic simple enough to do in your head:

total = TTFT + (output_tokens x inter_token_latency)

example: 0.8s + (600 tokens x 0.020s) = 0.8 + 12.0 = 12.8s

In that example TTFT is 6 percent of the total. Halve it and you save 0.4 seconds out of 12.8. Halve inter-token latency instead and you save 6. And yet TTFT is the number worth defending, because it is the only part of the wait where nothing is happening on screen.

What actually makes TTFT slow

Four things, roughly in order of how often they are the culprit.

1. Prompt length. Reading the prompt is a single parallel pass over all of it, so this scales with input size. Going from a 2,000 token prompt to a 40,000 token prompt is the most common cause of a TTFT that suddenly got worse, and it is usually self-inflicted by stuffing more context in.

2. Queueing. Under load your request waits for a slot before any work begins. This is invisible in your own testing and often dominant in production.

3. Reasoning effort. Models that think before answering can spend the entire thinking budget before emitting a visible token. TTFT measured against visible output can jump from under a second to tens of seconds with no other change.

4. Cold routing. First call to a less-used model or region can include setup that later calls do not pay.

On the second point, queue time and API rate limits are two views of the same capacity problem, and hitting a limit is just queueing that gave up.

The fix that actually works: stop re-reading the same prompt

If your prompt has a large stable prefix, a system prompt, a policy document, a schema, a few-shot block, the model is re-reading it on every request. Prompt caching stores the processed form of that prefix so subsequent requests skip the work. On long prompts this is the difference between seconds and hundreds of milliseconds, and it usually costs less per token as well. The mechanics and the cache-hit rules are in our explainer on prompt caching.

Two structural habits make caching work: put the stable material first and the variable material last, and stop editing the stable part casually. A one-word change near the top of a cached prefix invalidates everything after it.

Inter-token latency is a different lever

Once tokens are flowing, speed is set by how fast the hardware can read the model weights, one pass per token. You cannot prompt your way out of that. The techniques that help are architectural: smaller models, quantised weights, or speculative decoding, where a small model drafts several tokens and the large one verifies them in a single pass. The underlying reason a token costs a full read of the weights is covered in how a model actually produces a token.

Which number to optimise, by interface

That last row catches people out. An agent making eight sequential model calls pays TTFT eight times. A 900 millisecond TTFT that felt fine in a chat window becomes 7.2 seconds of dead time in a loop, before any actual generation.

Measuring it honestly

Report percentiles, not averages. TTFT distributions have long tails driven by queueing, and a mean hides exactly the requests that make users leave. Log p50, p95 and p99, tag each request with input token count, and you will usually find your p99 problem is a small number of enormous prompts rather than a platform issue.

Also measure from where the user is, not from your server. A TTFT of 300 milliseconds measured at the edge of your backend can be 1.2 seconds in a browser on a bad connection, and the browser is the number that counts.

Questions

What is a good TTFT?

Under 1 second feels responsive in a chat interface. Under 500 milliseconds is required for voice. Reasoning models routinely exceed 10 seconds before visible output, which is acceptable only if the interface says something is happening.

Does streaming reduce latency?

It reduces perceived latency, not total time. The same tokens arrive at the same moments; streaming just shows them as they arrive instead of holding them until the end. Mechanically it is server-sent events carrying incremental deltas, as Anthropic's streaming documentation sets out, so the transport does not change how fast the model produces anything.

Is TTFT affected by output length?

No. Output length affects total completion time. TTFT is settled before the second token exists. Setting a lower max output does not make the first token arrive sooner.

Why did my TTFT get worse without a code change?

Usually load on the provider side, a longer prompt fed by growing conversation history, or a cache that stopped hitting. Check input token count per request first, since it is the variable you own.

How does TTFT relate to tokens?

A token is the unit a model reads and writes, roughly three quarters of an English word. TTFT is the cost of reading all your input tokens and producing exactly one output token. The unit itself is explained in what is a token in AI.