What is inference in AI, and why it costs money

Inference in AI is what happens when a trained model answers a question. Training is the one-off process that produces the model's weights. Inference is every use of those weights afterwards: your prompt goes in, the model does arithmetic over billions of numbers, and a token comes out. Then it does the whole thing again for the next token. That repetition is why inference, not training, is the line item on your bill.

Most explanations stop at that distinction. The useful part is what makes inference expensive, because it is not what people assume.

What happens during inference in AI

Your text is split into tokens, roughly word fragments. Each token becomes a vector. Those vectors pass through the model's layers, and at each layer they are multiplied against matrices of weights. At the end the model produces a probability distribution over its vocabulary, picks one token, appends it to the sequence, and starts again. Providers bill this in exactly those units, which is why every published price list quotes a rate per million input and output tokens rather than per request.

Two phases, with very different characteristics:

This is why providers charge different rates for input and output tokens, often by a factor of three or four. They are not different products with different margins bolted on. They are genuinely different work.

The bottleneck is memory, not maths

Here is the part that explains the whole hardware industry right now. To generate one token, the chip has to read the model's weights out of memory and into the compute units. For a large model that is tens or hundreds of gigabytes moved, per token.

The arithmetic itself is trivial for a modern accelerator. The chip spends most of its time waiting for data to arrive. Decoding is memory-bandwidth bound, which means a GPU running inference is frequently idle in a way its utilisation graph does not show.

That single fact explains a lot of otherwise puzzling behaviour. Why batching helps: read the weights once, serve many requests with them. Why smaller models are disproportionately faster, not just proportionally. Why quantization is such a big lever, since halving the precision of the weights roughly halves the bytes you have to move. And why serious money is going into chips designed to sidestep the memory trip rather than into faster arithmetic.

Training versus inference, in the numbers that matter

Training is enormous, finite and someone else's problem unless you are a lab. Inference is small per request, unbounded in aggregate, and entirely yours.

A useful way to hold it: training cost is fixed and paid once, inference cost is variable and paid per user, per session, forever. A product with strong retention has an inference bill that grows with its success. That is a very different shape from hosting a web app, where the marginal cost of an extra page view rounds to nothing, and it is the thing most first-time AI product pricing gets wrong.

If you are sizing this for a real product rather than a mental model, how much does it cost to run an AI-built app puts numbers on it.

Levers that actually reduce inference cost

In rough order of how much they return for the effort:

1. Use a smaller model for the parts of the job that do not need a big one. Classification, extraction and routing almost never do. This is the biggest single win and the most consistently skipped.

2. Cut your input. Long system prompts and stuffed context are charged on every call. Most are half boilerplate nobody has re-read since the first week.

3. Cache. Prompt caching on repeated prefixes, and ordinary result caching on repeated questions, which is more common in real traffic than teams expect.

4. Cap output length. Decode is the expensive phase and models are verbose by default unless told otherwise.

5. Batch anything asynchronous. If a job does not need an answer this second, batched processing is often materially cheaper.

Notice that four of the five are about sending and receiving less rather than finding a cheaper provider. Provider shopping is the lever people reach for first and it is usually the smallest one.

Why inference hardware suddenly matters

Because decode is bandwidth-bound, there is an obvious if extreme fix: stop moving the weights. Put them in the silicon. That approach moved from research curiosity to acquisition target this month, when AMD agreed to buy Taalas, a startup whose chip has a single model etched into its metal layers at the foundry. The trade-off is exactly what you would expect. A chip with one model physically baked in can be dramatically faster and cheaper for that model, and useless for any other. It cannot be reprogrammed when the next model ships.

For anyone building on APIs this is not a decision you make, it is a reason to expect inference prices to keep falling in steps rather than smoothly. Prices drop when a new generation of inference hardware lands, then sit still. Plan pricing on today's rates, and treat the drops as margin rather than as a strategy.

FAQ

Is inference the same as running a model?

Yes. Inference is the industry term for using a trained model to produce output. Running, serving and inference all describe the same activity.

Why is output more expensive than input?

Input is processed in one pass. Output is generated token by token, each one requiring another read of the model's weights. The work per output token is simply larger.

Does a bigger context window cost more?

Yes, and it compounds, because attention cost grows faster than linearly with sequence length. Long context is a real capability with a real bill attached, as what is a context window covers in more detail.

Can I run inference locally to avoid the cost?

You can, and it moves the cost rather than removing it: hardware, electricity and your time. For small models on a machine you already own it can work well. Running an AI coding model locally goes through what is realistic.

Inference cost is one input into a bigger question builders keep getting wrong: does a bigger AI context window mean better answers looks at the other side of that cost, what you actually get for it.