What Is an Attention Sink in AI Models?

An attention sink is a token that a model pours attention onto without using it for meaning, usually the first few tokens of the sequence. It exists because attention has to add up to one. When no earlier token is genuinely relevant, that weight has to land somewhere, and models learn to dump it on the opening tokens. The practical consequence is strange: drop those apparently useless tokens from the cache and output quality collapses, which is exactly what happens when a naive sliding window runs long enough.

Why the surplus has to go somewhere

In self-attention, every token computes scores against the tokens before it, and those scores are normalised so they sum to one. That normalisation is the whole story. A token that genuinely needs nothing from its predecessors still cannot assign zero attention overall, because the total is fixed.

So the model needs a parking space, and the first tokens make an ideal one. Every position can see them, since attention only looks backwards, and their content is usually generic. The model learns during training to send surplus attention there. The authors of the paper that named the effect found these scores are strong toward initial tokens even when those tokens carry no semantic importance.

The failure this explains

The reason anyone outside model research cares is a specific, observable break.

To hold a conversation longer than the context window allows, the obvious approach is a sliding window: keep the most recent tokens in the KV cache and discard the oldest. You would expect the model to forget early details and otherwise carry on.

That is not what happens. Once the window slides past the very first tokens, output degrades sharply, into repetition and incoherence rather than graceful forgetting. The sink was evicted, the surplus attention has nowhere to go, and the distribution across the remaining tokens distorts. The model is not missing information so much as miscomputing with what it still has.

The fix in Efficient Streaming Language Models with Attention Sinks (Xiao et al., ICLR 2024) is almost comically cheap: keep the first few tokens' cached states permanently, alongside the sliding window of recent ones. Four initial tokens were enough. With that in place the authors report stable language modelling across up to four million tokens on several model families with no fine-tuning, and large speedups over recomputing the window each time.

Keeping four tokens is not keeping the conversation

This is the point most summaries get wrong. Retaining sink tokens restores numerical stability. It does not restore memory. The content that slid out of the window is gone, and the model cannot refer to it. What you get is a model that stays coherent while having genuinely forgotten the middle of the conversation, which is a better failure than incoherence but still a failure.

It is also unrelated to the separate problem of long conversations getting slower, which comes from the cost of attending over a growing cache and is covered in why models slow down in long conversations.

From an accident to a design choice

Sinks began as an emergent quirk that researchers noticed. They are now something model architects build in deliberately. The paper itself suggested adding a dedicated placeholder token during pretraining so the model has an explicit sink rather than borrowing whichever token happens to be first.

That idea has reached shipping models. The published model card for Recursive's Flint, covered in our write-up of that release, lists learned per-head attention sinks among its attention components, alongside 35 sliding-window layers and 14 full-attention layers. That combination is the point: a model leaning heavily on windowed attention for efficiency needs the sink mechanism to be reliable, so it is specified rather than hoped for.

Our explainer on sliding window attention covers the efficiency trade that makes all of this worth doing, and our guide to how AI models work sets the mechanism in context.

What this means if you are building

Mostly, it means a known problem is already handled for you. Serving frameworks and modern architectures account for sinks, so you are unlikely to meet the raw failure through a hosted API.

It matters if you implement your own context trimming. Writing a loop that drops the oldest messages to stay under a limit is the same mistake at the application layer, with the same result when the system prompt goes out of the window. Keep the opening anchored, and summarise the middle rather than deleting from the front.

Frequently asked questions

Why do models attend to the first token so heavily?

Because attention scores are normalised to sum to one, so surplus attention must go somewhere. The earliest tokens are visible to every later position and usually carry little meaning, which makes them a convenient place to park it.

Does keeping attention sinks extend the context window?

No. It keeps output coherent while the window slides, but the content that scrolled out is still gone. Stability and memory are separate things.

How many sink tokens are needed?

The StreamingLLM authors found four initial tokens sufficient to restore stable behaviour with window attention.

Is an attention sink the same as a system prompt?

No. A system prompt is instruction content the model reads for meaning. A sink is a position that absorbs attention weight regardless of its content. A system prompt sitting at the start may happen to serve both roles.