Latency vs Throughput in AI: Which to Optimise

Latency vs throughput in AI comes down to one question: are you optimising for how fast one answer arrives, or for how many answers you can produce per minute? Latency is the wait for a single request. Throughput is volume across many. They pull in opposite directions, because the batching that raises throughput also makes any individual request wait longer. Pick the wrong one and you either pay for capacity nobody notices or ship something that feels broken while your dashboards look fine.

The three numbers that actually decide it

"Latency" is too vague to act on. Three measurements split it into things you can change:

• Time to first token (TTFT). How long from sending the request to the first piece of output appearing. Dominated by prompt processing, so it grows with prompt length.

• Tokens per second (TPS). How fast output arrives once it has started. Roughly constant for a given model and setup.

• Requests per minute, or per-request concurrency. How many of these you can run at once before the queue starts adding wait.

The first two are latency. The third is throughput. A user reading a streamed answer feels TTFT most, then TPS. A nightly job processing four thousand records feels only the third.

Why the two trade against each other

Inference splits into two phases, covered in more depth in prefill and decode. Prefill reads your prompt and is compute-heavy. Decode generates output one token at a time and is memory-bandwidth-heavy, which leaves the hardware underused on a single request.

The fix for that waste is batching: run many requests through the model together so each pass through the weights serves more than one user. Throughput climbs steeply. But a request that arrives just after a batch starts has to wait for a slot, and once batched it shares bandwidth with everything else in the group. Its TTFT and TPS both get worse.

That is the whole trade. Larger batches mean more total work done and a longer wait for any given piece of it. Batch inference takes this to its conclusion: no interactivity at all, maximum volume, often at a discount.

A decision table for four app shapes

The third row is the one teams get wrong. An agent doing twelve tool calls feels slow, so the instinct is to chase latency on each call. Usually the win is bigger from running steps four through seven at the same time, because the user is waiting on the sum, not on any single request.

Streaming is a latency fix that costs nothing

The cheapest improvement to perceived latency is not a faster model. It is showing output as it is generated instead of waiting for the whole response. Anthropic's API documents this directly: you set "stream": true and the response arrives incrementally using server-sent events, with separate deltas for text, tool use and thinking.

Total time to the final token is unchanged. Time until the user sees something drops to TTFT. For a 600-token answer at 50 tokens per second, that is the difference between a twelve second blank screen and half a second followed by text the user can read while the rest arrives.

Two things to get right when you stream. Render partial output as it lands rather than buffering it, or you have paid the engineering cost for none of the benefit. And handle the pause: when a model calls a tool mid-turn, the stream goes quiet while the tool runs, and an interface with no indicator for that reads as a hang.

Where the numbers go wrong in practice

Measuring from the wrong place. Vendor latency figures are server-side. Your users add TLS setup, network distance and your own middleware. Measure from the browser.

Averaging. A 400ms mean TTFT with a 3 second 95th percentile is a product where one request in twenty feels broken. Track p95 and p99 or you will not see it.

Long prompts quietly taxing every request. TTFT scales with prompt length, so a system prompt that grew from 300 to 4,000 tokens over three months has slowed every single call. Prompt caching removes most of that cost for the stable part.

Throughput measured without the deadline. Rows per hour is meaningless unless you know when the job has to finish. Work backwards from the deadline, then buy the smallest concurrency that meets it.

The one case where you get both

Speculative decoding, explained in what is speculative decoding, uses a small fast model to draft tokens that the large model then verifies in a single pass. When the draft is right, several tokens land for the cost of one. It raises TPS without larger batches, which is why it is the rare optimisation that helps interactive latency and total throughput at the same time. It is a serving-stack feature, so it is available to you only if you self-host or your provider has enabled it.

Beyond that, assume you are choosing. Decide which number your users actually feel, instrument that one, and let the other drift. For the underlying mechanics of what the hardware is doing during each phase, how AI models work covers the ground this post stands on.

FAQ

Is lower latency always better?

No. Below the threshold where a human notices, further latency reduction buys nothing, and it usually costs throughput or money. For a chat interface, anything under about a second to first token is already in the "feels instant" range.

Does a bigger model always mean higher latency?

Higher per-token latency, generally yes, because there is more computation per token. But a larger model that answers correctly in one attempt can beat a small model that needs three tries, so compare end to end rather than per call.

How do I measure time to first token properly?

Timestamp the moment you send the request and the moment the first content delta arrives, client-side, and record the difference per request. Then look at the distribution, not the mean. Time to first token goes through the details.

Can I improve throughput without more hardware?

Often yes. Shorter prompts, caching the stable prompt prefix, capping max output tokens, and moving work that does not need interactivity to a batch endpoint all raise effective throughput on the capacity you already have.