Local AI Models: What They Are and When to Use

Local AI models are models whose weights sit on hardware you control, so inference happens on your machine and no request leaves it. That is the entire distinction. Everything else people attach to the term, that local models are more private, cheaper, weaker, or harder to run, is a consequence of that one property, and each consequence has become less true over the past year.

The useful question is not whether local is better. It is which of your workloads belong on a machine you own, and which belong on somebody else's.

How local AI models differ from hosted ones

The version stability row is underrated. A hosted model can be retired with a few months of notice, and your prompts can drift before that. A local model does exactly what it did last year, forever, which matters more for a product feature than for a chat window.

What you can actually run

Model size is measured in parameters, and memory requirement roughly tracks it, reduced by quantization. Quantization stores weights at lower precision, trading a small amount of accuracy for a large amount of memory, which is what makes consumer hardware viable at all. The mechanics are covered in what quantization is in AI.

As a rough map of the current landscape:

• Up to 8B parameters: runs on most modern laptops. Good for summarising, classifying, extracting fields, and simple drafting.

• 13B to 30B: needs a 16GB to 24GB GPU or an Apple silicon machine with a large unified memory pool. This band now covers competent tool use and coding.

• Above 30B: workstation or rented GPU territory. The gap to hosted frontier models narrows here but does not close.

The 30B band is where the interesting movement has been. Meta's Apache 2.0 release, covered in Meta Muse Glimmer, lists a 24GB quantized build, which is a single mainstream consumer card. Catalogues like the Ollama library are a reasonable way to see what is currently available at each size.

When local AI models are the right call

Four situations where the answer is usually yes:

1. The data cannot leave. Client records under an agreement, health or legal material, anything where a data processing addendum is harder to get than a GPU.

2. Volume is high and predictable. This is where the arithmetic turns, and it is worth doing rather than assuming.

3. You need the model to never change. Regulated workflows, reproducible pipelines, anything you have validated once and do not want to revalidate.

4. You are offline or on a poor connection by default.

On the arithmetic: a 24GB consumer card is a few hundred to low four figures, plus power. Against that, a typical small-model API call costs fractions of a cent. Break-even often lands somewhere in the tens of thousands of requests per month, and below that the API wins comfortably. Run your own numbers rather than trusting a general claim, using your real request volume and your real electricity price. The same reasoning applies to reducing AI API costs generally.

Where local still loses

Hardest reasoning tasks, very long context work, and anything needing the absolute best available quality still go to hosted frontier models. Local models are also work: you own the setup, the updates, the memory management, and the debugging when a quantized build behaves subtly differently from the full-precision one. That labour is real and it is easy to leave out of a cost comparison.

There is also a licensing dimension that catches people. Open weights does not automatically mean you may ship the output commercially, and the terms vary sharply between releases. That is covered in AI model licences.

FAQ

Do local AI models keep my data private?

Inference is private, because nothing leaves the machine. That is not the same as your setup being secure, which depends on where the weights came from and what the model is allowed to touch.

Do I need a GPU?

For models under about 8B, no. CPU inference is slow but workable. Above that a GPU or Apple silicon with generous unified memory stops being optional.

Are local models much worse than hosted ones?

At the same parameter count, no. The gap is a size gap, not a quality gap: you can run 30B at home and not 500B, and the difference between those shows up on hard reasoning more than on everyday extraction and drafting.

Can I fine-tune a local model?

Yes, and it is one of the stronger arguments for running locally, though it needs more memory than inference alone. Start with what fine-tuning is before deciding it is what you need.

Most teams end up mixed rather than pure: a local model for the high-volume, privacy-sensitive, boring work, and a hosted model for the hard cases. That split is easier to reason about once you know how AI models work at the level of what is actually being computed.