Run an AI Coding Model Locally: What You Need

To run an AI coding model locally you need enough GPU memory to hold the weights plus the context, and the arithmetic is simple enough to do before you buy anything. A model's weights in memory are roughly the parameter count multiplied by the bytes per parameter. At 16-bit precision that is 2 bytes each, at 8-bit quantisation 1 byte, at 4-bit roughly half a byte. A 27-billion-parameter model therefore needs about 54GB at full precision, 27GB at 8-bit, and 14GB at 4-bit, before you add anything for context.

That last clause is where most estimates go wrong, so here is the whole calculation.

The memory arithmetic

Three things occupy GPU memory when you run a model:

The KV cache is the attention key and value tensors for every token currently in context. It scales linearly with how much context you are actually using, and for a coding model that is the whole point of running it, because you want it to see your files. A long session with a large repository in context can add many gigabytes on top of the weights.

Practical rule: take the quantised weight size, add 20 to 30 per cent for realistic context, add 2GB, and compare that to your card. If the result does not fit with room to spare, you will either run at trivial context lengths or spill into system memory, at which point speed collapses.

What that means for real hardware

Apple Silicon changes the shape of this because memory is unified: a machine with 64GB of shared memory can load models that would need a very expensive discrete GPU. It is slower per token than an equivalent-memory NVIDIA card, but it can hold things that card cannot, and for a single user waiting on a response that trade is often fine.

What quantisation actually costs you

Quantisation reduces the precision of each weight. 4-bit is roughly a quarter the memory of 16-bit and the quality loss is small enough that it is the default for local use. It is not free, though, and coding is one of the places it shows.

The degradation is not uniform. Short completions and boilerplate hold up well. What degrades first is long-range consistency: keeping a naming convention across a large file, remembering a constraint you set twenty exchanges ago, correctly threading a type through several functions. Those are exactly the tasks people want a coding model for.

So the honest framing is that 4-bit is the right default and you should expect to notice it on hard problems. Going to 8-bit if you have the memory is usually worth more than moving up a size class at 4-bit.

What local actually gets you

Three things, and it is worth being precise about which of them you need.

Privacy that is structural rather than contractual. Nothing leaves the machine. For work under a confidentiality clause that forbids third-party processing, this is the difference between compliant and not. That is a real answer to a real problem, and it is the strongest reason on this list. Is it safe to give AI access to my data covers when this matters.

Cost that is fixed rather than metered. After the hardware, inference is electricity. If you run high volume, this changes the economics. If you run a few dozen requests a day, it does not, and the hardware will not pay for itself.

Availability that does not depend on anyone else. No rate limits, no deprecations, no outages, no policy changes. Works on a plane.

What local costs you

Capability. A 27B model at 4-bit is not competitive with a frontier API model on hard reasoning. It is competitive on completion, refactoring, explaining code and writing tests. Match the tool to the task and it is fine. Expect it to architect a system for you and you will be disappointed.

Your time. Driver versions, runtime updates, quantisation formats, models that load but produce nonsense because the prompt template is wrong. This is genuine ongoing overhead, and it is the cost most people fail to price in.

The context ceiling. This is the underrated one. API models advertise very large context windows; local models are constrained by the KV cache arithmetic above. Feeding a large codebase to a local model is exactly the operation your memory budget makes hardest. What a context window actually is explains why this constrains what you can ask.

The point where local stops being worth it

A clear line, since most guides avoid drawing one.

Local is worth it if any of these is true: you are contractually barred from sending code to a third party, you run enough volume that metered inference is a real line item, you need it to work offline, or you enjoy the tinkering enough that the maintenance is not a cost.

Local is not worth it if your main motivation is saving money on a modest amount of usage, or if you expect it to match frontier-model quality on architecture and debugging. On both counts the numbers do not support it. A mid-range GPU costs many months of typical API usage, and the quality gap on hard tasks is not closed by quantisation tricks.

A reasonable middle path that few people consider: run a small local model for the high-frequency, low-stakes work, completion and boilerplate and test generation, and keep an API for the hard problems. The local model handles the volume, the API handles the cases where quality decides the outcome.

The near-term thing worth watching is what happens to the self-hostable tier. Qwen3.8's open-weights announcement includes a 27B checkpoint alongside the flagship, and 27B at 4-bit is squarely inside what a 24GB card handles. That size class is where local inference either becomes broadly practical or does not.

Getting started without buying anything

Before spending money, test the premise on what you already own. Any recent laptop can run a 7B model at 4-bit slowly, which is enough to answer the only question that matters: does a model at this capability level actually help with your work, or were you imagining a frontier model on your desk?

Run it for a week on real tasks. Note how often you gave up and opened an API tool instead. That ratio, not a benchmark table, tells you what to buy. What an AI benchmark measures is worth reading first if you are tempted to shop by leaderboard, and the broader landscape is covered in our guide to AI coding tools.

Deciding to self-host is one migration path among several, weighed in how to know when to upgrade to a newer AI model

Running a model locally usually means running a compressed version of it. See what quantization actually does to understand the tradeoff.

FAQ

What is the minimum GPU to run an AI coding model locally?

Around 8GB of VRAM will run a 7B model at 4-bit precision with modest context. It works, but it is the level at which you are mostly getting smart autocomplete rather than something that reasons about your codebase.

Is a local model as good as a cloud API model?

Not on hard reasoning tasks. On completion, refactoring, writing tests and explaining existing code, a well-chosen 14B to 32B model is close enough that the difference rarely decides anything.

Does quantisation ruin code quality?

4-bit quantisation costs some long-range consistency, which shows up as forgotten constraints and drifting conventions in longer sessions. Short completions are largely unaffected. 8-bit is noticeably more reliable if you have the memory.

Can I run a model on a Mac?

Yes, and unified memory makes larger models viable than the equivalent price in discrete GPU. Generation is slower per token, which matters less than you would think for interactive single-user work.

How much context can I actually use locally?

Less than the model's advertised maximum, because the KV cache competes with the weights for the same memory. Budget 20 to 30 per cent of your weight size for context and verify with your actual workload rather than trusting the specification.