What Is RAG? Retrieval-Augmented Generation Explained

What is RAG in AI? Retrieval-augmented generation is a technique that connects a large language model to an external source of documents, so it can pull in specific, current facts before writing an answer instead of relying only on what it memorized during training. Instead of guessing from patterns learned months or years ago, the model looks something up first, then generates a response grounded in what it just found. That single change is why RAG has become the default architecture for chatbots and internal search tools that need to answer questions about a specific company's documents rather than the general internet.

How RAG actually works

RAG splits the job into two stages that run every time a user asks a question: retrieval, then generation. The quality of the final answer depends almost entirely on what gets retrieved in the first stage.

The technique was formalized in a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, the lab now known as FAIR within Meta Superintelligence Labs,, which combined a pretrained language model's built-in memory with a searchable index of external text. Their finding: models that could look things up produced answers that were more specific and factual than models generating from training alone.

• Retrieval: the system converts the user's question into a numerical representation (an embedding) and searches a document index for the passages that match most closely, usually the top 3-8 chunks.

• Augmentation: those retrieved passages get inserted directly into the prompt sent to the language model, alongside the original question.

• Generation: the model writes its answer using the retrieved text as grounding, ideally citing or paraphrasing only what's actually in those passages.

As Amazon Web Services describes it, RAG augments a model with external data so it has the context it needs to produce accurate output for a specific use case, rather than relying purely on whatever was baked into its training run.

Worked example: a support bot that reads a PDF policy doc

Here's what that looks like end to end: a small business wants a chatbot to answer customer questions from a 12-page PDF return policy, instead of someone answering the same five questions all day.

Step 1: Chunk and index the document

Before any customer asks anything, the PDF gets broken into chunks, roughly paragraph-sized pieces, maybe 200-500 words each. Each chunk is converted into an embedding (a vector of numbers representing its meaning) and stored in a vector database alongside the original text.

Step 2: A customer asks a question

A customer types: "Can I return a swimsuit if I opened the packaging?"

Step 3: Retrieval

The system embeds that question and compares it against every chunk in the index. It pulls back the two or three chunks closest in meaning, likely a paragraph about hygiene-restricted categories and one about the standard 30-day return window. Chunks about shipping costs or gift cards get left out because they scored lower on relevance.

Step 4: Augmented prompt

Those retrieved paragraphs get stitched into a prompt sent to the model: "Using only the following policy excerpts, answer the customer's question. Policy excerpts: [retrieved text]. Customer question: Can I return a swimsuit if I opened the packaging?"

Step 5: Answer

The model reads the actual policy language, not a guess, and answers: swimwear is final sale once the hygiene seal is broken, per the retrieved clause. The answer is traceable back to a specific paragraph in the PDF, which matters if the business ever needs to check what the bot told someone.

What breaks when you skip RAG

Without retrieval, the same bot is just a general-purpose language model with no idea what this business's policy says. Asked the swimsuit question, it will produce something plausible-sounding, because IBM notes that models without grounding fall back on patterns learned during training rather than the specific facts of a situation. It might invent a 14-day window that doesn't exist, state a policy for the wrong product category, or make up an exception outright. None of that is malicious, the model has no way to know it's wrong, and it says it with the same confidence as a correct answer.

That gap is why RAG exists: it doesn't make a model smarter, it gives it something real to read first. IBM also notes this doesn't eliminate hallucination entirely, a model can still misread a retrieved passage, but it closes the biggest hole, which is answering from nothing at all.

RAG vs. other ways of adding knowledge

For a deeper side-by-side comparison of RAG, fine-tuning, and long-context prompting, see how each approach handles cost, latency, and source traceability differently.

Where RAG fits with related concepts

RAG usually shows up alongside two other pieces of AI infrastructure. One is the context window, the amount of text a model can hold in a prompt at once, which caps how many retrieved chunks you can pass in. The other is the Model Context Protocol, a newer standard for how a model connects to external tools and data, which some RAG systems now use as the plumbing between retriever and model.

If you're building this kind of assistant yourself, the general process of building an app with AI covers the broader steps, and automating customer support with AI walks through the support-bot use case specifically, including where a RAG setup like the one above saves the most staff time.

RAG solves the knowledge half of a common pairing. The other half, teaching a model consistent behavior rather than new facts, is fine-tuning, which works on a fundamentally different problem than retrieval does.

Frequently asked questions

Is RAG the same as fine-tuning?

No. Fine-tuning changes a model's weights through additional training, good for teaching tone or format but unreliable for teaching specific facts. RAG leaves the weights untouched and feeds it relevant text at answer time, which makes updates easy: edit the source document, not retrain a model.

Does RAG stop AI hallucination completely?

No. It removes the most common cause, the model having no relevant information at all, by giving it real text to work from. A model can still misread or misquote a passage it retrieved correctly, so RAG lowers the error rate without eliminating it.

What kind of documents can RAG use?

Any text that can be split into chunks and embedded: PDFs, help center articles, product manuals, internal wikis, spreadsheets converted to text, transcripts. The PDF policy document above is a typical starting point for a small business.

Do I need a vector database to build RAG?

For anything beyond a handful of documents, yes, or something that behaves like one. It stores the embeddings and runs the similarity search that finds the right chunks quickly, which matters once you're indexing more than a few dozen pages.

How is RAG different from just asking ChatGPT to search the web?

Web search plugins are one form of retrieval, pointed at the open web instead of a private document set. The mechanics match: search, retrieve, feed into the prompt. RAG usually refers to the pattern applied to an organization's own documents, where the index is something the business controls.

RAG systems pull your own documents into the prompt, which raises the same hygiene question as any other AI workflow. See how to prompt AI without leaking sensitive data for what to redact before it goes into the index.