What Is Quantization in AI? Smaller Models Explained

What is quantization in AI? It is storing a model's weights at lower numeric precision than they were trained at, typically dropping from 16 bits per weight to 8, 5 or 4. A model with 7 billion parameters occupies roughly 14GB at 16-bit precision and roughly 4GB at 4-bit. That is the whole idea: the same model, described less precisely, small enough to fit somewhere it previously did not.

What is actually being compressed

A trained model is a very large collection of numbers. Each weight is stored in a format with a fixed number of bits, and more bits means finer distinctions between values. Quantization maps those numbers onto a coarser scale, in much the same way that rounding prices to the nearest pound loses information while keeping the shopping list usable.

The important property is that the compression is not uniform in its effects. Some weights matter enormously to the output and some barely matter at all, which is why modern quantization schemes are not a flat rounding operation. They allocate more precision to the parts of the network that are sensitive to it and less to the parts that are not, and that asymmetry is the difference between a 4-bit model that works and one that produces nonsense.

The numbers that matter

Those figures are approximations that ignore the memory needed for context, which grows with how much text you feed the model and can add several gigabytes on long inputs. Budget for the weights plus the context, not the weights alone. The llama.cpp project's quantization documentation lists the exact formats and their measured size and quality trade-offs, and is the reference worth reading if you are choosing between specific levels.

Why quantization often makes models faster

The intuitive explanation, that smaller numbers are quicker to multiply, is the less important one. Generating text one token at a time is usually limited by memory bandwidth rather than arithmetic, meaning the hardware spends most of its time moving weights rather than computing with them. Halve the size of the weights and you halve the amount of data that has to move for every token.

This is why quantization can produce a speedup that looks disproportionate to the compression, and why the effect is strongest on hardware with limited memory bandwidth, which describes most laptops.

What you lose

Quality loss from quantization is real, uneven and easy to miss in casual testing. It tends to show up first in the places where precision matters most.

• Long chains of reasoning, where a small error early compounds through subsequent steps.

• Exact reproduction, meaning quoting text, generating precise formats, or producing valid structured output.

• Rare knowledge, which appears to be stored more diffusely and is more easily damaged by coarse rounding.

• Instruction following at the margins, where a model becomes slightly more likely to ignore a constraint you stated once.

Everyday conversational tasks are the least affected, which is exactly why quantized models feel fine in a quick test and then disappoint in production. Test on your hardest real task, not your easiest one. If your workload depends on machine-readable output, the failure modes in getting JSON output from AI reliably get noticeably more common at aggressive quantization levels.

Choosing a level

A workable default: start at 4-bit, because it is where the size saving is largest relative to the quality cost. Move up to 5-bit or 8-bit if you can see the difference on your own task. Go below 4-bit only when the model would not otherwise fit at all, and expect to notice.

The more useful comparison is often not between quantization levels but between models. A larger model quantized aggressively frequently outperforms a smaller model at full precision for the same memory budget, which is a more interesting decision than picking a bit width. Small language models covers the other end of that trade-off, and running an AI coding model locally is where most people meet quantization for the first time in practice.

None of this applies when you call a hosted API, where the provider has already made these choices for you and does not usually publish them. That opacity is one of the quieter arguments for open-weight models, where the precision you run at is a decision you get to make and can reproduce.

FAQ

Does quantization make a model dumber?

Slightly, and unevenly. At 8-bit the difference is hard to measure on most tasks. At 4-bit it is measurable but usually acceptable. Below 4-bit it becomes visible in ordinary use, particularly on reasoning and exact formatting.

Can I quantize a model myself?

Yes. Tools in the llama.cpp ecosystem convert a released model into quantized formats, and most popular models already have community-quantized versions published, which saves you the conversion step entirely.

Is a quantized 13B model better than a full-precision 7B model?

Frequently yes, at a similar memory footprint, though it depends on the task. It is worth testing both on your own workload rather than assuming, because the answer varies more by task type than most benchmarks suggest.

Does quantization affect training or only inference?

The kind described here applies to inference, meaning running an already-trained model. Quantization during training exists and is a different and more involved topic, aimed at reducing training cost rather than deployment size.