What Is FlashAttention? Why Long Context Got Cheap

FlashAttention is a way of computing attention that moves less data between a GPU's memory levels. It does not approximate anything: the output is numerically the same as standard attention. The speedup comes entirely from reorganising memory traffic, which is a strange thing for an algorithmic advance to be, and it is the reason million-token context windows went from impossible to routine.

If you have wondered why context windows grew by orders of magnitude without a corresponding jump in hardware, this is a large part of the answer.

The problem it solves

Attention compares every token in a sequence against every other token. For a sequence of length n, that is an n-by-n matrix of scores. At 100,000 tokens, that matrix has ten billion entries.

The textbook implementation builds that matrix, writes it to memory, reads it back to apply softmax, writes the result, reads it again to multiply by the values. Each of those steps moves the full matrix between the GPU's main memory and its compute units.

Here is the thing that makes this the bottleneck: a modern GPU can do arithmetic far faster than it can fetch the numbers to do arithmetic on. The chip spends most of attention waiting for data to arrive. The matrix multiplication itself is not the expensive part. Reading and writing that enormous intermediate matrix is.

The two kinds of GPU memory

A GPU has a memory hierarchy, and the gap between levels is enormous.

HBM is the big pool the model weights sit in. SRAM is the small, very fast scratchpad attached to the processing units. Standard attention repeatedly round-trips the score matrix through HBM because the matrix is far too large for SRAM.

What FlashAttention does instead

It never builds the full matrix.

The algorithm splits the sequence into blocks small enough to fit in SRAM, loads a block of queries and a block of keys, computes that tile of scores on-chip, applies softmax to it, multiplies by the corresponding values, and accumulates the running result. Then it moves to the next block. The score matrix exists only one tile at a time, in fast memory, and is never written to HBM at all.

The subtlety is softmax, which normally needs to see an entire row before it can normalise anything. FlashAttention handles this by keeping a running maximum and a running sum per row and rescaling the accumulated output as new blocks arrive. This is an online softmax, and it is what makes the tiling exact rather than approximate.

The original paper, FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness by Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra and Christopher Ré, frames the contribution as making attention IO-aware: accounting for reads and writes between levels of GPU memory as a first-class cost rather than counting arithmetic operations. The reference implementation is open source, and some version of it is what most inference stacks actually run.

The word "exact" is doing a lot of work

Plenty of techniques make attention cheaper by computing less of it: sparse attention that skips token pairs, low-rank approximations, sliding windows that only look at nearby tokens. All of them trade quality for speed, and all require you to check whether the trade hurt your use case.

FlashAttention does not make that trade. Same inputs, same outputs, same model quality, less time and less memory. That is why it was adopted essentially universally rather than as an option you evaluate. There is no downside column.

It also explains why you have probably never configured it. It is not a model feature, it is an implementation of an operation your model already performs, and it is on by default in every serious inference stack.

What it changes for you

Memory scales linearly, not quadratically. Standard attention needs memory proportional to n². FlashAttention needs memory proportional to n. This is the change that made long contexts feasible at all: the quadratic term was what made 100,000 tokens impossible, and it is gone.

Long context still costs, just differently. Compute is still quadratic in sequence length even when memory is not, so a longer prompt is still a more expensive prompt. FlashAttention removed a wall, not the slope. Why long context costs more covers what you are actually billed for.

It does not help with the KV cache. During generation, the model stores keys and values for every token processed so far, and that cache grows linearly and lives in HBM regardless. FlashAttention speeds up the attention computation; it does not shrink the KV cache. That is why memory pressure during long generations is a separate engineering problem with separate solutions.

Prefill gets much faster than decode. Processing a long prompt is a big parallel operation that FlashAttention accelerates dramatically. Generating tokens one at a time is a different shape of work and benefits far less. This asymmetry is part of why time to first token and per-token throughput behave so differently as prompts grow.

The general lesson

The interesting thing about FlashAttention is not the tiling. It is that a substantial speedup in the most studied operation in machine learning came from thinking about memory movement rather than about mathematics.

That pattern keeps recurring. Quantisation, paged attention, speculative decoding, and continuous batching are all, in different ways, about moving less data or keeping the expensive hardware busy. The arithmetic was rarely the constraint.

For how attention works in the first place, start with self-attention, which covers the mechanism this optimises. For how far it stretches in practice, what is a context window is the companion piece, and the how AI models work guide ties the pieces together.

FAQ

Does FlashAttention change model output?

No. It computes exact attention, producing the same result as the standard implementation. Only the memory access pattern differs.

Do I need to enable FlashAttention?

Generally not. It is the default in current inference and training stacks, since there is no quality cost to weigh against the speedup.

How much faster is it?

It depends heavily on sequence length and hardware, because the benefit comes from avoiding memory traffic that grows quadratically. The longer the sequence, the larger the gain, which is why it mattered most for long-context models.

Does it reduce the memory a model needs?

It removes the quadratic memory needed for the intermediate attention matrix, making that part linear in sequence length. It does not reduce model weights or the KV cache.

Is FlashAttention the same as sparse attention?

No. Sparse attention skips computing some token pairs and is approximate. FlashAttention computes every pair and is exact.