What Is Model Collapse in AI?

Model collapse is what happens when a generative model is trained on data produced by earlier generative models, generation after generation, without fresh human-made material mixed in. The rare cases at the edges of the distribution disappear first, the output narrows towards the average, and eventually the model produces confident nonsense. It is a real, demonstrated effect under specific conditions. It is not a description of what is happening to the assistant you used this morning.

Where the term comes from

The phrase entered general use with a 2024 Nature paper by Ilia Shumailov and colleagues, AI models collapse when trained on recursively generated data. The researchers trained a model, generated data from it, trained the next generation on that output, and repeated. Within a handful of generations the models degraded in a consistent way, and the paper's central conclusion was that indiscriminate use of model-generated content in training causes irreversible defects in which the tails of the original distribution disappear.

Two words in that sentence do the heavy lifting. Indiscriminate, meaning no filtering and no fresh human data. And tails, meaning the unusual cases rather than the common ones.

What breaks first

Not accuracy, at least not initially. Variety.

• Rare cases vanish. Uncommon names, unusual sentence structures, minority dialects and edge cases stop appearing, because each generation samples fewer of them than the last.

• Output converges on the average. Everything drifts towards the most probable phrasing, which reads as blandness long before it reads as error.

• Confidence stays high. The model does not signal that it has lost coverage. It answers narrow questions with the same certainty as before.

• Then accuracy goes. Once the distribution has narrowed far enough, later generations start producing content unmoored from anything real.

The mechanism resembles photocopying a photocopy. Each pass is a faithful copy of the previous pass rather than of the original, so small losses compound. It is worth distinguishing this from what an AI hallucination is, which is a property of a single model's output rather than a degradation across training generations.

What the paper does not say

The finding has been stretched a long way past its evidence, so it is worth being precise about the limits.

• It does not say that current commercial models are collapsing. They are not trained recursively on unfiltered output from themselves.

• It does not say synthetic data is harmful. Curated synthetic data, filtered and mixed with human data, is a routine and effective part of modern training.

• It does not say the effect is inevitable. A follow-up analysis of the paper's assumptions argued that the result depends on each generation fully replacing the previous data rather than accumulating alongside it, and that collapse is substantially avoided when data accumulates.

• It does not predict a date. Claims that models will collapse by a particular year are not in the paper and are not supported by it.

Does model collapse affect the model you use today?

Not measurably, and the reason is that labs treat this as an engineering problem they already know about. Training data is filtered, provenance is tracked, human data is deliberately retained and weighted, and synthetic data is generated for specific purposes rather than scraped indiscriminately. A quality drop you notice between two versions of a product is far more likely to come from a changed system prompt, a different default tier, or a safety adjustment than from anything happening in the training corpus.

There is a genuine long-term concern underneath, which is that the open web now contains a great deal of machine-generated text and separating it from human writing is getting harder. That is a data-sourcing problem for the people building the next generation of foundation models, and it is why licensing deals for human-produced archives have become worth so much. It also sits one level below how AI models work at all, which is the useful place to start if any of the vocabulary here is new.

When it becomes your problem

If you generate training data with a model and then train on it, the risk stops being theoretical. Three habits keep it manageable:

• Keep human data in the mix and know its share. If you cannot say what proportion of your set is machine-generated, that is the first thing to measure.

• Filter rather than accept. Score generated examples and discard the weak ones. The Nature result is about indiscriminate use, and discrimination is the entire defence.

• Hold out a fixed human-written evaluation set. If your test data is also generated, the narrowing happens on both sides and your numbers will look fine while quality falls.

This applies most directly to anyone doing fine-tuning on generated examples, which is now a common and often sensible approach, and to a lesser extent to model distillation, where a smaller model learns from a larger one's outputs on purpose. Distillation works precisely because it is discriminate: a specific teacher, a specific task, curated data, one generation rather than an unbounded loop.

Questions people ask

Is AI-generated content on the web already causing this?

It makes the data-sourcing job harder rather than causing collapse directly. Labs filter aggressively and license human archives specifically because of this, which is a response to the risk rather than evidence of the outcome.

Should I avoid using synthetic data entirely?

No. Synthetic data is useful and widely used, particularly for covering rare cases you have no real examples of. The failure mode is unfiltered recursion, not synthesis, and the difference between the two is curation.

Would I notice model collapse if it happened?

Not from casual use, because the first symptom is narrower variety rather than visible error. You would notice it in a fixed evaluation set that includes unusual cases, which is another argument for keeping one. The same evaluation habit also catches the knowledge cutoff quietly shifting under a model you rely on.

It is also a real consideration when a provider ships a new version. how to migrate from one AI model to another covers testing for regressions like this before you fully switch over.