How Much VRAM Do You Need to Run a Local LLM?

You can estimate the memory a local model needs with one line of arithmetic: parameters in billions, times bits per weight, divided by 8, gives the size of the weights in gigabytes. A 7B model at 4-bit is about 3.5 GB, and the same model at 16-bit is 14 GB. Add roughly a fifth on top for overhead, plus room for the context. That total is the VRAM you want, and if it does not fit, you either shrink the model or accept a much slower setup.

The formula

Weights memory in GB is parameters in billions multiplied by bits per weight, divided by 8. Eight bits is one byte, so an 8-bit model needs one gigabyte per billion parameters. Real quantised files come out a little larger than the nominal figure because they also store scaling data, so treat the result as a floor.

Those are my own arithmetic from the formula, not benchmark results. If quantisation is new, what quantisation is explains how a model shrinks without being retrained.

The two things the formula leaves out

• Overhead. The runtime needs working memory beyond the weights. A rule of thumb is to add around 20 percent.

• The context. Every token in the conversation adds to a cache of keys and values that lives in the same memory. It grows as your chat or document gets longer, which is why a model that loads fine can run out of memory mid-task. What a KV cache is covers why.

A rule of thumb for your GPU

Keep the weights under about 60 percent of your VRAM, which leaves space for overhead and a reasonable context. At 4-bit, that works out like this.

So a 32B model at 4-bit, with 16 GB of weights, is tight even on a 24 GB card once the context grows. A 70B model at 4-bit needs about 35 GB for the weights alone and will not fit on any single consumer card.

Mixture-of-experts models are different

A mixture-of-experts model uses only a few of its sub-networks for each token, so it computes quickly for its size. It still has to store all of them. A 125B model at roughly 2 bits per weight is about 31 GB of weights by the same formula, which is why the Strata project's 12 GB GPU claim depends on 32 to 64 GB of system RAM as well, as the Strata README lists. Read what mixture of experts means for the idea, and plan memory by total parameters, not active ones.

What to do when it does not fit

1. Pick a smaller quantisation, for example 4-bit instead of 8-bit.

2. Shorten the context you ask the model to hold.

3. Choose a smaller model. A well-matched 14B often beats an overstretched 32B you can barely run.

4. Offload some layers to system RAM, accepting a large drop in speed.

A five-step check before you download

1. Find the parameter count on the model page.

2. Choose the quantisation you plan to use.

3. Apply the formula and add 20 percent.

4. Add room for the context length you need.

5. Compare with your VRAM. Under 60 percent of it is comfortable.

For choosing between running locally and paying for an API, see local AI models and running an AI coding model locally. The fundamentals are in how AI models work.

Frequently asked questions

Can I run a local LLM on a CPU only?

Yes, using system RAM instead of VRAM, but generation is much slower. It works for experiments and short answers, and it is frustrating for long tasks.

Does more system RAM help if my VRAM is too small?

It lets you load a model that does not fit in VRAM by offloading part of it, at a large cost in speed. The more of the model sits in RAM, the slower each token arrives.

Is 8 GB of VRAM enough?

It is enough for models up to roughly 9B at 4-bit with a modest context, using the 60 percent rule above. That covers many useful small models.

How much VRAM does a coding model need?

The same formula applies. Coding sessions tend to use long contexts because you paste in files, so budget more headroom than for short chat.