What Is Context Rot in AI? Why Long Chats Get Worse
Longer prompts don't just risk hitting a token limit, they quietly make models less reliable long before that. Here's what causes context rot and how to work around it.
Context rot in AI is the steady decline in a language model's accuracy as the input it has to process gets longer, even when that input stays well inside the model's stated context window. Add more tokens to a prompt, whether that's chat history, pasted documents, or tool output, and answer quality drops. Not because the words fall out of memory, but because the model's attention spreads thinner across everything it's holding. The Chroma Research study that named the phenomenon tested 18 frontier models, including GPT-4.1, Claude, Gemini 2.5, and Qwen3, and found every one got measurably worse as input length grew, sometimes by 30 to 50 percent, well before hitting documented context limits.
How context rot in AI differs from hitting the context window limit
The two get conflated because both are measured in tokens, but they're not the same thing. A context window is a hard capacity limit, the maximum tokens a model can hold in one request. Context rot is a soft, gradual decline in retrieval accuracy that starts well before that limit. A model with a 200,000-token window might handle a 5,000-token prompt cleanly and start missing details or pulling the wrong item from a pasted list by the time the conversation reaches 60,000 tokens. The window tells you what fits. It says nothing about how reliably the model uses what's in it.
What the research actually found
Chroma Research published the first large-scale study of this effect in 2025, running controlled retrieval and text-replication tasks across 18 frontier models. Degradation isn't uniform: GPT models tend to hallucinate confidently once confused, while Claude models more often refuse to answer rather than guess. Coherent, well-written documents were sometimes harder to search accurately than randomly shuffled chunks, because coherent prose hands the model more plausible-sounding distractors.
The pattern lines up with earlier work. The 2023 Stanford paper Lost in the Middle: How Language Models Use Long Contexts found that models retrieve information best when it sits at the start or end of a prompt, and worst when buried in the middle, a primacy-and-recency bias that persists even in long-context models. That's a big part of why an AI forgets things in long chat sessions: the fact usually isn't gone, it's just sitting in the part of the context the model attends to least.
Why more tokens doesn't mean equally reliable recall
Attention in a transformer weighs every token against every other token, so as a prompt grows, the model's effective focus per token gets diluted. That's the mechanical reason larger inputs hurt context window performance even when nothing is technically full. Similarity decay compounds it: the more the correct answer resembles surrounding text, the faster accuracy collapses as length increases. For the underlying mechanics of how models weigh and predict from input tokens, it helps to understand how AI models actually process input.
Practical ways to keep long sessions reliable
None of these eliminate context rot, but each reduces how much irrelevant material the model has to weigh at once.
Strategy | What it does | Best used for |
|---|---|---|
Chunking | Retrieve only the relevant slice of a document instead of pasting the whole thing | Long reference docs, codebases, knowledge bases |
Periodic summarization | Compresses older turns into a short summary and drops the raw transcript | Long-running chats or agent sessions |
Fresh sessions | Starts a new conversation once a task shifts, instead of extending one indefinitely | Multi-stage projects, coding agents, task pivots |
Chunking and retrieval work together: instead of dumping a fifty-page spec into a prompt, pull the two or three sections that actually answer the question. That's the core idea behind context engineering, treating prompt input as curated rather than dumped. Coding agents need the same discipline for tool output and file reads, covered in this guide on keeping a coding agent from losing context across long sessions.
Summarization needs the same care. Asking a model to compress its own transcript works, but the compression can lose exactly the detail you need later unless you specify what to keep. A practical guide on prompting an AI to summarize a long document covers the structure that keeps a summary useful instead of generic.
Signs context rot is happening in a live session
The model repeats or contradicts something it already said earlier in the same chat
It answers a question you already covered by guessing instead of recalling the earlier answer
It grabs the wrong item from a list you pasted, especially one buried in the middle
It starts ignoring formatting or instructions that were given early in the session
If two or more of these show up at once, the fix isn't a bigger context window. It's shortening what's in play: summarize, trim, or start over.
Frequently asked questions
What causes context rot in AI models?
It comes down to how transformer attention works. As input grows, the model has to weigh more tokens against each other, diluting how sharply it can focus on any single fact. Chroma Research and the earlier Lost in the Middle study both found this shows up as accuracy that drops well before the context window is technically full.
Does a bigger context window fix context rot?
No. A larger window raises the ceiling on how much text you can fit in, but it doesn't change the attention mechanics that make recall less reliable as length grows. Chroma's tests found measurable degradation in models with context windows as large as 1 million tokens.
How many tokens before context rot kicks in?
There's no fixed threshold, it depends on the model and task. Chroma's research found degradation well before most models hit their documented limits, and other researchers point to a noticeable drop around 300,000 to 400,000 tokens for some 1M-token models. Simple tasks and shorter prompts hold up longer than complex, multi-step ones.
Can you prevent context rot in a long AI chat?
You can't eliminate it, but you can delay it. Chunking input to what's relevant, summarizing older turns instead of keeping a full transcript, and starting a fresh session when a task changes direction all reduce how much redundant context the model has to weigh at once.
Is context rot the same thing as hitting the context window limit?
No. Hitting the limit means the input doesn't fit and gets truncated or rejected. Context rot is the gradual accuracy decline that happens while the input still fits comfortably, caused by degraded attention rather than a hard cutoff.
How did this land?
About the author
Senior Editor, AI & Product
Cecilia leads the Swarmz editorial desk. She has spent a decade turning complex AI and product topics into writing people actually finish, and she owns the blog's quality bar.
