How Much GPU Memory Does an AI Model Need?

How much GPU memory an AI model needs comes down to one multiplication plus one addition that most guides leave out. The multiplication: number of parameters times bytes per parameter. The addition: the KV cache, which grows with how much context you are processing and how many requests you serve at once. Get the first and you know whether the weights fit. Get the second and you know whether the model is actually usable.

The short version for a 7-billion-parameter model: about 14GB at half precision, about 4GB heavily quantised, plus a gigabyte or two of cache. The arithmetic is the same at every size, which is what makes it worth learning.

The formula for how much GPU memory an AI model needs

Weights first. Every parameter is a number, and the precision you store it at decides how many bytes each one costs:

• fp16 or bf16, half precision: 2 bytes per parameter. The usual default for serving.

• int8: 1 byte per parameter. Roughly half the memory, small quality cost on most tasks.

• int4: about 0.5 bytes per parameter. A quarter of the memory, more noticeable quality cost.

So a 7B model at fp16 is 7,000,000,000 times 2 bytes, which is 14GB. The same model at int4 is about 3.5GB. That is the whole trick behind quantization: it does not shrink the model, it stores each number in fewer bits.

Then add the KV cache. During generation the model keeps the keys and values for every token it has processed so far, so it does not recompute them for each new token. That cache scales with sequence length and with the number of simultaneous requests. On a 7B model a single short conversation might add a few hundred megabytes; long documents across many concurrent users can add more than the weights do.

This is the term that catches people. A model that loads fine on your card and then runs out of memory halfway through a long document has not got bigger. Its cache has. The relationship between context length and resource cost is the same one described in our explainer on context windows, seen from the memory side instead of the billing side.

Worked example: a trillion-parameter model

Mistral published the open weights for a roughly 1-trillion-parameter model in late October 2026. Run the arithmetic:

• At fp16: 1 trillion times 2 bytes is about 2,000GB, or 2TB of memory for the weights alone.

• At int8: about 1TB.

• At int4: about 500GB.

A high-end data-centre GPU carries 80GB to 192GB. Even at int4, that model needs several of the largest cards available, networked together, before you have spent anything on the KV cache. Which is the practical lesson: open weights means you are allowed to download it and permitted to run it. It does not mean you can.

Meanwhile a 2-billion-parameter model is about 4GB at fp16 and runs on a laptop. The range of things called "open weight models" spans three orders of magnitude of hardware requirement, and the licence tells you nothing about which end you are at.

Quick reference

Approximate memory for the weights, before cache:

• 2B parameters: 4GB at fp16, 1GB at int4. Laptop territory.

• 7B to 8B: 14 to 16GB at fp16, 4GB at int4. One consumer GPU.

• 70B: 140GB at fp16, 35GB at int4. One large card quantised, or two to four consumer cards.

• 400B+: multiple data-centre GPUs regardless of quantisation.

• 1T: a small cluster.

Add 20 to 40 percent on top for the cache and runtime overhead if you are serving more than one request at a time. If the result is close to your card's capacity, treat it as not fitting. A model that just barely loads will fail on the first long input.

Why this matters before you commit

The memory number decides the cost question, not the other way round. People start from "self-hosting should be cheaper" and work toward hardware; the useful direction is to compute the memory requirement, see what hardware that implies, price that hardware, and then compare against API pricing. Frequently the comparison stops being interesting at the first step. The break-even analysis in the self-hosting break-even calculation assumes you have already established the model can run on something you can afford.

It also explains a pattern worth recognising. When a lab releases a very large model's weights, the practical audience is other labs, serving providers and researchers, not individual builders. The releases that change what one developer can do are the small ones, and they get a fraction of the attention.

Common questions

Can I split a model across several GPUs?

Yes. Inference frameworks shard a model across cards, and across machines for the largest models. It works, and it adds interconnect bandwidth as a new bottleneck, which is why multi-GPU serving is slower per card than single-GPU serving.

Does the KV cache grow with output length or input length?

Both. It covers every token the model has processed, prompt and generated output together. A long prompt costs cache memory before the model writes anything.

Is CPU RAM a substitute for VRAM?

Partly, and slowly. Most runtimes will offload layers to system memory when VRAM runs short. The model works; throughput drops by an order of magnitude or more.

Why do published memory requirements differ from this arithmetic?

Published figures usually assume a specific quantisation, a short context and a single request. The formula gives you the floor, and real serving sits above it.