NPU vs GPU for AI: What Runs Where, and Why

A GPU is a wide, general parallel processor with a lot of memory bandwidth. An NPU is a narrow processor built for the specific matrix and convolution operations neural networks use, optimised for work per watt rather than peak throughput. For AI, the short version is: GPUs train and serve almost anything, NPUs serve a narrower set of shapes more cheaply, and the gap between them is closing on inference and not on training.

The comparison usually gets framed as speed. It is more useful to frame it as which bottleneck you are up against, because for most language model work the bottleneck is not arithmetic at all.

The dividing line is memory bandwidth

Generating a token from a transformer means reading the model weights out of memory, multiplying them against a small activation, and writing a result. With a batch size of one, you read every weight to produce a single token. The arithmetic per byte read is terrible. The processor spends most of its time waiting for memory.

That has a blunt consequence: for single-stream generation, tokens per second is roughly memory bandwidth divided by model size in bytes. A 14 GB model on a part with 900 GB/s of bandwidth cannot exceed about 64 tokens per second no matter how fast its maths units are. Adding compute to that situation buys nothing.

Training is the opposite. You process large batches, reuse each weight across many samples, and the arithmetic per byte goes up by orders of magnitude. Now the maths units are the constraint, and the wide, high-precision, well-interconnected part wins. This is why the training and serving markets are pulling apart into different silicon, and why our explainer on what inference actually means keeps landing on the same point: serving is a different workload wearing the same name.

NPU vs GPU for AI, side by side

The weakness row is the one people underestimate. An NPU is fast on the operations its compiler knows how to map. Hand it a model with an unusual attention variant or a custom kernel and the toolchain either falls back to the CPU or refuses. GPUs degrade gracefully in that situation. NPUs do not.

Where NPUs actually show up

Three places, and they are less exotic than the phrase suggests.

1. In the device you are reading this on. Phone and laptop NPUs run wake-word detection, image processing, background blur, on-device transcription and increasingly small language models. The win is battery life, not speed.

2. In data centre inference accelerators, as a deliberate alternative to buying more GPUs for serving. This is the segment attracting the money right now.

3. In embedded and industrial hardware, where the power envelope is fixed and the model rarely changes.

If you are experimenting with running models yourself, the NPU in a recent laptop is the most accessible version of this, and our guide to running AI models locally covers what is realistic there. The honest summary: small models run pleasantly, large ones do not, and the constraint is memory again.

Why TOPS numbers mislead

NPU marketing leads with TOPS, trillions of operations per second. It is a peak arithmetic figure, measured at the lowest precision the part supports, with every unit busy. For a memory-bound workload it predicts nothing. Two parts with identical TOPS and a 3x bandwidth difference will differ by close to 3x on single-stream token generation.

The numbers that do predict something:

• Memory bandwidth in GB/s, which sets the ceiling on tokens per second for small batches.

• Usable memory capacity, which sets the largest model you can hold without spilling.

• Supported precisions, because a part that only does INT8 forces you into quantised weights.

• Compiler and runtime coverage, which decides whether your specific model runs at all.

On that third point, quantisation is not a free lunch, and the trade-offs are laid out in what quantisation does to a model. A part that is fast only at INT4 is fast at a version of your model that is measurably worse.

How batching changes the answer

Everything above assumes you are serving one request at a time. Batch several requests together and each weight read serves many tokens, arithmetic intensity climbs, and the workload starts looking like training. That is why hosted APIs are cheap per token and your local single-user setup is not: they are amortising the same memory read across dozens of users. The mechanics are in our explainer on batch inference, and it is the single largest cost lever in serving, ahead of hardware choice.

So which should you care about?

If you are calling an API, neither. You are buying tokens, and the hardware behind them is the provider's problem. Where it surfaces is in price and rate limits, both of which move when serving economics move.

If you are running models yourself, budget for memory bandwidth and capacity first and treat compute as the thing you will not run out of. If you are shipping software to end-user devices, an NPU is worth targeting when the model is small, stable and runs constantly, and not worth it otherwise. The broader mechanics of how any of this executes sit in how AI models work.

Questions

Is an NPU faster than a GPU?

Not usually in absolute terms. A data centre GPU will out-throughput a device NPU by a wide margin. The NPU wins on work per watt within its supported operations, which is a different question.

Can an NPU train a model?

Most cannot, and the ones that can are limited. Training needs higher precision, more memory and different communication patterns. NPUs are built for the forward pass.

Do I need an NPU to run AI locally?

No. A recent CPU with fast memory runs small models acceptably, and a consumer GPU runs medium ones well. The NPU mainly helps battery life on laptops and phones.

What is a TPU then?

A TPU is an NPU by another name, built by Google for its own data centres, sitting at the large end of the category rather than the device end. The architectural principle is the same: narrow the operation set, win efficiency.