What Happens When AI Runs Out of Context Window

When a conversation or document exceeds a model's context window, one of three things happens, and which one depends on how the application is built, not on the model itself. Some APIs reject the request outright with a hard error. Some client code or agent frameworks quietly drop the oldest messages to make room for new ones, a pattern called sliding-window truncation. And some systems run middleware that compresses or summarizes older turns before they get cut. The model has no memory beyond what's actually sent in that request, so whatever gets left out of the prompt effectively never happened, as far as the model is concerned.

What a context window actually limits

A context window is the maximum number of tokens a model can process in a single request, input and output combined. Every message, every document you paste in, every tool call and tool result counts against that budget, which is one piece of the bigger picture covered in how AI models work. The number varies a lot by model and vendor, and it has grown substantially across model generations, but every model has some ceiling. The question worth asking isn't what the limit is, it's what happens the moment you cross it.

Three concrete outcomes at the moment of overflow

1. Hard rejection. Many raw API calls simply refuse a request that's too large. You get a client error back, no completion generated. This is the cleanest failure mode because it's loud: your code sees an error and can retry with less input.

2. Sliding-window truncation. This is the quiet one. A lot of chat applications keep sending the full conversation history with every request, and once that history would exceed the budget, the app or the SDK's convenience wrapper drops the oldest turns to fit. Nothing errors. The API call succeeds. The model just never sees the beginning of the conversation anymore.

3. Summarization or compression middleware. More sophisticated agent frameworks compress older turns into a condensed summary instead of dropping them outright, then feed that summary back in as compact context on later turns. This preserves the gist of earlier turns at the cost of detail and precision.

The symptom that gives it away

If you've built anything with a long-running chat session or agent loop, you've probably seen this: the assistant confidently contradicts something it said forty messages ago, or ignores an instruction you gave right at the start of the session, something like always respond in bullet points or never suggest a specific feature. It looks like the model got confused or stopped following directions. Usually it's simpler than that: that early instruction fell out of the window. It was truncated or summarized away, and the model is now operating on a version of the conversation that literally doesn't contain it anymore.

This is a distinct failure from context rot, where the model's output quality degrades within a context that still technically fits, simply because long, dense, or noisy context makes retrieval from context less reliable. Overflow is about content that is gone from the prompt. Context rot is about content that's present but poorly used. They produce similar-looking symptoms, the AI "forgetting" something, but the fix for each is different, so it's worth not conflating them.

It's also worth separating this from prompt caching, which is unrelated to overflow. Prompt caching is a cost and latency optimization: reusing the computed representation of a prefix you've already sent, so a long system prompt or document doesn't get reprocessed from scratch on every call. It has nothing to do with what happens when your input exceeds the window. It just makes staying under the window cheaper and faster.

Why this sneaks up on builders

Most people don't hit this on day one. It shows up after a session or document grows past what fit comfortably during testing. A support bot that's fine for a five-turn exchange starts dropping the customer's original problem statement by turn thirty. A "chat with your PDF" feature that worked on a ten-page contract breaks silently on a two-hundred-page one, because the whole document got stuffed into a single prompt instead of being processed in pieces. An agent that appends every tool call's raw output to its running context fills its own window with logs and JSON blobs long before the actual task is done. In every case the app kept working, technically. It just started working on less information than the user thought it had.

Four ways to handle it today

• Chunk documents instead of stuffing them whole. Split long input into sections, by page, heading, or a fixed token count, and process or query each chunk separately rather than concatenating an entire document into one prompt.

• Retrieve instead of stuffing everything into context. Store chunks in a vector index or search system, and pull only the handful most relevant to the current question into the prompt. This keeps the working context small regardless of how large the underlying source material is, and it's the standard fix for the "chat with your documents" case.

• Periodically re-state key instructions. If there's a rule the assistant absolutely must follow for the whole session, don't rely on it surviving a hundred turns of truncation. Re-inject it: pin it in a system message that gets resent every request, or have your app re-insert it every few turns so it's never far enough back to get cut.

• Summarize old turns before you're forced to. Rather than letting truncation silently drop the earliest messages, have your application proactively roll up older turns into a short summary once the conversation passes a length threshold, and replace the raw turns with that summary. You control what gets kept, instead of finding out by accident. This is also where a well-designed KV cache and caching strategy matter for keeping the process fast and affordable as conversations grow.

None of these require a bigger context window model, and reaching for one only postpones the problem. A bigger window costs more per request, runs slower, and doesn't fix the version of this that shows up as context rot instead of overflow. Fixing the context management is worth doing regardless of which model sits behind it.

Frequently asked questions

How do I know if my chatbot hit its context window limit?

Watch for two signatures: an explicit API error mentioning input length or a token limit, or a subtler pattern where the assistant stops honoring instructions or facts from early in a long session even though nothing crashed. The second one means truncation or summarization happened quietly rather than failing loudly.

What's the difference between context window overflow and context rot?

Overflow means content was removed from the prompt, truncated or summarized away, because there wasn't room for it. Context rot means the content is still in the prompt, but the model's quality degrades anyway because the context is very long, dense, or cluttered. Overflow is a capacity problem; context rot is a quality problem that can happen even when you're nowhere near the limit.

Does truncating older messages delete them permanently?

Not necessarily. Most applications keep the full conversation in a database and only trim what gets sent to the model on each request. The history isn't destroyed, it's just no longer part of what the model can see unless your app retrieves and re-inserts it.

Will using a model with a much larger context window fix this?

It raises the ceiling, so overflow happens later, but it doesn't remove the failure mode, and very large contexts bring their own tradeoffs in cost, latency, and the separate issue of context rot. Deliberate context management, chunking, retrieval, periodic re-statement, summarization, still pays off even on models with huge windows.