What Is Model Distillation? A Plain Explanation
Distillation is why the cheap tier of every model family got surprisingly capable. A big model teaches a small one, and the small one inherits judgement it was too small to learn alone.
Model distillation is training a small model to imitate a large one. The large model, called the teacher, produces outputs for a set of inputs; the small model, called the student, is trained to match them. The student ends up better than a model of its size trained the ordinary way, because it is learning from a source that has already worked out which answers are plausible, not just which one is correct.
It is the main reason the cheap tier of every model family became genuinely usable rather than a toy.
How model distillation actually works
The obvious version of distillation would be: run the big model on a pile of questions, keep its answers, train the small model on those question-and-answer pairs. That helps, and it is not the interesting bit.
The interesting bit is that the teacher does not output one answer. At every step it produces a probability across every possible next token. Ask it to classify a photo of a spaniel and it might say 91 percent dog, 6 percent wolf, 2 percent cat, and a long tail of almost nothing. Those small numbers carry information: the teacher has learned that a spaniel is more confusable with a wolf than with a cat.
Training the student against that whole distribution, rather than against the single label "dog", transfers the teacher's sense of what is nearly right. This is the insight from the original distillation paper by Hinton, Vinyals and Dean in 2015, which named the effect dark knowledge: the useful signal is hiding in the answers the teacher rejected and how firmly it rejected them.
A small model trained on hard labels has to rediscover all of that structure from scratch, with far less capacity to do it in. A distilled student is handed it. That is a different failure mode from model collapse, where a model degrades because it was trained on unfiltered AI-generated output rather than guided by a deliberate teacher-student setup.
How it is done in practice
Choose the teacher and the student architecture. The student is usually the same family, five to fifty times smaller.
Generate the training signal. Run the teacher across a large set of inputs, keeping the full output distribution where possible and the generated text where not.
Train the student to match it, typically against a mix of the teacher's distribution and the real labels where real labels exist.
Evaluate on the tasks you care about, not on average benchmark scores, because distillation loses capability unevenly.
That fourth step is where teams get surprised. A distilled model can hold nearly all of the teacher's performance on common tasks and lose a lot on rare ones, because rare cases were thinly represented in whatever the teacher was run over. Average scores hide this completely.
How it differs from the other shrinking techniques
Technique | What it changes | When you use it |
|---|---|---|
Distillation | Trains a new, smaller model to imitate a bigger one | You want a permanently cheaper model with most of the behaviour |
Quantization | Stores the same model's numbers at lower precision | You want the same model to fit in less memory, right now |
Pruning | Removes weights or whole components judged unimportant | You have a specific size or latency target to hit |
Fine-tuning | Adjusts an existing model on task-specific data | You want different behaviour, not a different size |
They compose. A production small model has often been distilled from a larger one and then quantized to run on cheaper hardware. Quantization is the one most often confused with distillation, and the distinction is simple: quantization keeps the same model and describes it less precisely, distillation builds a different, smaller model that behaves similarly. Fine-tuning is a different axis again, since it changes what a model does rather than how large it is.
Why it matters if you are building something
You will almost certainly never distil a model yourself. It needs the teacher's internals, serious compute, and a large input set. But you consume the results constantly, and knowing what you are consuming changes two decisions.
First, model selection. The small, fast, cheap tier in a model family is usually a distilled sibling of the flagship rather than a separate lineage. That is why it feels like the big model on ordinary work and falls apart on unusual work, and it is why benchmark parity between tiers should be treated with suspicion. Test the small tier on your actual edge cases before switching, since that is exactly where distillation loses the most, and an average benchmark score will not warn you.
Before switching tiers, it also helps to check what the provider actually disclosed about that model's evaluated behaviour, which is exactly what reading an AI model's system card walks through.
Second, cost. Distilled models are the single largest reason inference prices have fallen, and moving suitable workloads down a tier is usually the biggest available saving. Classification, extraction, routing and summarisation of short inputs are the usual candidates, and they cover a surprising share of what most applications actually do. The rest of the levers are in reducing AI API costs.
The limits, and the licence question
Distillation cannot create capability the teacher does not have, and it reliably loses some the teacher does. Long-horizon reasoning, rare languages and unusual formats degrade first, and often invisibly, because the student produces a fluent answer rather than an obviously broken one.
There is also a legal dimension that has become contentious. Most commercial model providers prohibit using their outputs to train competing models, which makes distilling a closed frontier model a terms-of-service matter rather than a purely technical one. Distilling from a model you are licensed to use that way, or from open-weight models under a permissive licence, avoids the question. The difference between open-weight and closed models is what decides which route is available to you.
Common questions
Is a distilled model always worse than its teacher?
On overall capability, yes. On a specific narrow task it can match or beat the teacher, because the training concentrated its limited capacity where you pointed it.
How much smaller can the student be?
Ten times smaller with modest loss is common. Beyond about fifty times the losses get severe and uneven, though the exact point depends heavily on how narrow the target task is.
Is this the same thing as a small language model?
Not quite. Distillation is a training method; a small language model is a size category. Most good small models were distilled, but a small model can also be trained from scratch.
Can you distil reasoning ability?
Partially, and it is an active area. Training a student on a teacher's full chain of reasoning rather than only its conclusions transfers noticeably more than answer-matching alone, which is a practical illustration of the general point that how a model works is mostly a story about what it was trained to predict.
How did this land?
About the author
Senior Editor, AI & Product
Cecilia leads the Swarmz editorial desk. She has spent a decade turning complex AI and product topics into writing people actually finish, and she owns the blog's quality bar.
