What Is Chunking in RAG? A Practical Explainer
Chunking decides what your retriever can find. A worked example of one document split three ways, the failure each split causes, and how to size chunks.
Chunking in RAG is the step where you cut source documents into smaller passages before storing them, because retrieval returns chunks, not documents. Get it wrong and the symptom is specific and maddening: your system confidently answers a question with information that is almost right, while the correct sentence sits in your files untouched. The model is not hallucinating. The retriever never handed it the passage, because the passage was cut in half.
That makes chunking the highest-leverage and least glamorous decision in a retrieval pipeline. Here is what it does, what goes wrong, and how to size it without guessing.
One document, three splits, three different failures
Take a 40 page employee handbook and a user asking: how many days of unpaid leave can I take after two years?
Split it per page, and the answer table lands on page 12 while the sentence defining what counts as two years of service sits on page 11. Retrieval finds the table, the model reads a column header without its definition, and answers for the wrong tenure band.
Split it into 100 token pieces, and you get a chunk that reads, in full: 'After 24 months, up to 15 days.' Retrieved on its own, nothing in it says leave, unpaid, or handbook. Its embedding sits nowhere near the question, so it is never retrieved at all.
Split it by section heading, and the leave policy stays intact with its definitions, its table and its exceptions. The chunk is 700 tokens, longer than the tidy default, and it works, because the unit of meaning was a section, not a page or a token count.
That is the whole lesson. Chunk boundaries should follow the document's own structure, and only fall back to arbitrary sizes when there is no structure to follow.
How chunking fits the rest of the pipeline
Each chunk gets converted into a vector and stored. At query time the question becomes a vector too, and the store returns the nearest chunks. If you have not met that machinery yet, our explainer on RAG covers the flow end to end, and what an embedding actually is explains why a chunk with no context words scores badly.
The important consequence: a chunk is the smallest thing your system can retrieve and the largest thing it can retrieve. Both halves bite. Too small and the passage loses the context that makes it findable. Too large and you spend context window on padding, and precision drops because one relevant sentence drags four irrelevant paragraphs along with it.
Chunking strategies, and when each one is right
Strategy | How it splits | Best for | Fails on |
|---|---|---|---|
Fixed size | Every N tokens, ignoring content | Unstructured text dumps, transcripts, a fast first version | Tables, lists and anything where a boundary lands mid-thought |
Recursive | Tries paragraph, then sentence, then character, until it fits | Mixed prose. The sensible default for most projects | Documents whose meaning spans sections, like contracts with cross-references |
Structural | On headings, list items, table rows, code blocks | Handbooks, docs, wikis, anything with real headings | Scanned PDFs and exports where the structure was lost |
Semantic | Splits where the topic shifts, measured by embedding distance | Long unstructured writing where topics drift | Cost and complexity. You embed twice, and boundaries get hard to debug |
Start structural if your documents have structure. Fall back to recursive if they do not. Reach for semantic only after you have measured that the simpler options are the bottleneck.
Sizing: sensible starting numbers
Common vendor guidance lands in a similar range. Weaviate's guide to chunking strategies and the practitioner write-up from Unstructured both point at a few hundred tokens as a starting point rather than a rule.
Start around 400 to 512 tokens, roughly 1,600 to 2,000 characters of English.
Lean smaller, 256 to 512, when questions are factual lookups with one right answer.
Lean larger, 512 to 1,024, when questions are analytical and need surrounding argument.
Overlap of 10 to 20 percent is the usual default. It is insurance against a bad boundary, and it costs storage, so treat it as a knob to test rather than a setting to copy.
Two rules that matter more than any of those numbers. Keep the source title and section heading inside the chunk text, not only in metadata, so the embedding carries the words a user would actually type. And never let a table split across chunks: keep it whole, or repeat its header row in each piece.
Testing chunking without a research budget
You do not need an evaluation framework to catch the big failures. You need twenty real questions and the passage that should answer each.
# 20 questions, each with the doc section that truly answers it
hits = 0
for question, expected_section in test_set:
chunks = retrieve(question, k=5)
if any(expected_section in c.metadata["section"] for c in chunks):
hits += 1
print(f"recall@5: {hits}/{len(test_set)}")Run it, change one variable, run it again. If recall@5 is poor, chunking or embedding is your problem and no amount of prompt work will fix it. If recall@5 is strong but answers are still wrong, the right chunk is being retrieved and ranked below the noise, which is a reranking problem instead. Separating those two questions saves days.
Frequently asked questions
What is the best chunk size for RAG?
There is no single best size, but 400 to 512 tokens is a defensible starting point for mixed prose. Move smaller for factual lookups and larger for analytical questions, then measure recall on your own documents rather than inheriting someone else's number.
Do I still need chunking with a long context window?
Usually yes. A large window lets you pass more chunks, not skip retrieval. Stuffing entire document sets into every request is slow, expensive, and tends to bury the relevant passage. The trade-off is laid out in RAG versus fine-tuning versus long context.
Should chunks overlap?
Overlap of 10 to 20 percent is a reasonable default and protects against a boundary landing mid-sentence. It is not free: storage and index size grow, and near-duplicate chunks can crowd out variety in your results. If your splits already follow headings, you may need very little.
How do I chunk tables and code?
Treat both as atomic where you can. A table row without its header is unreadable to a retriever and to a model, so repeat the header if you must split. Code is similar: split on function or class boundaries, never on a fixed character count, or you will retrieve half a function.
Does chunking affect cost?
Yes, in both directions. Smaller chunks mean more vectors to store and more embedding calls at index time. Larger chunks mean more tokens per request at query time, which is the recurring cost. Since retrieval runs on every question and indexing runs occasionally, chunk size usually matters more to your bill than chunk count. The underlying mechanics are covered in how AI models work.
How did this land?
About the author
Senior Editor, AI & Product
Cecilia leads the Swarmz editorial desk. She has spent a decade turning complex AI and product topics into writing people actually finish, and she owns the blog's quality bar.
