What is a RAG (retrieval-augmented generation) system?

RAG is a design that combines a language model with an external, searchable knowledge base so the model answers from retrieved documents instead of from memory alone. The paper that named it describes "retrieval-augmented generation (RAG) — models which combine pre-trained parametric and non-parametric memory for language generation," where "the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index." In practice a RAG system splits your documents into chunks, embeds each chunk as a vector, stores those vectors in an index, and at query time retrieves the most relevant chunks and pastes them into the model’s prompt as context. It exists because a model has a fixed knowledge cutoff and cannot see your private corpus; retrieval closes both gaps without retraining.

When should you use RAG instead of fine-tuning?

Use RAG when the problem is knowledge — facts that change, that are private to you, or that must be cited. Retrieval lets you add, update, or delete a document by re-indexing it in seconds, and it lets the answer point at the source. Use fine-tuning when the problem is behavior — a tone, a format, a skill, or a domain style the model should internalize. The two are not rivals: many production systems fine-tune for form and retrieve for facts. The honest rule is that fine-tuning teaches the model how to say things, and RAG gives it something true to say.

Why is chunking so important in a RAG system?

Because retrieval can only return whole chunks, so the chunk boundary decides whether an answer is retrievable at all. Chunks that are too small split a fact across a boundary — no single chunk contains the whole answer, so retrieval returns a fragment. Chunks that are too large bury the relevant sentence in unrelated text, which dilutes the embedding and drops the chunk’s match score, so it loses to sharper chunks. There is a sweet spot in between, and adding overlap between adjacent chunks buys back facts that straddle a boundary. It is the cheapest lever with the biggest effect on retrieval quality, which is why it is worth tuning against real questions rather than guessing.

What is reranking and why add a second retrieval stage?

Reranking is a second, slower, more accurate pass over the candidates the first stage retrieved. The first stage — a vector index — is fast because it compares a query vector to millions of document vectors independently, but that independence is exactly why it is imprecise. A reranker is a cross-encoder that reads the query and one candidate together and scores their true relevance, which is far more accurate but far too expensive to run over the whole corpus. So you retrieve maybe 100 candidates cheaply, rerank them to pick the best 5, and feed only those to the model. The cost is latency: reranking N candidates adds N model calls, which is the trade you dial against your latency budget.

How do you evaluate a RAG system?

By measuring retrieval and generation separately, because they fail differently. Retrieval is scored with recall — did the chunk containing the answer make it into the retrieved set? — against a golden set of questions with known answers. Generation is scored on faithfulness (also called groundedness): is every claim in the answer actually supported by the retrieved context, or did the model make something up? A common framing is the RAG triad: context relevance (are the retrieved chunks relevant to the question), groundedness (is the answer supported by those chunks), and answer relevance (does the answer actually address the question). A high final answer with low retrieval recall is luck, not a working system, so you watch both.

Design a RAG System

A model with a knowledge cutoff and no access to your private documents has to answer questions about both — so you retrieve the right passages and hand them to it as context. We size the corpus in bytes and vectors, dial an interactive chunker until retrieval hits, draw the vector index and reranker honestly, and end on the part everyone skips: how you measure whether any of it worked. Drawn, computed, and animated.

System design · AI / ML. The source ↗

A free, interactive, animated visual explainer of Design a RAG System — built to be understood, not skimmed.

Questions

What is a RAG (retrieval-augmented generation) system?: RAG is a design that combines a language model with an external, searchable knowledge base so the model answers from retrieved documents instead of from memory alone. The paper that named it describes "retrieval-augmented generation (RAG) — models which combine pre-trained parametric and non-parametric memory for language generation," where "the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index." In practice a RAG system splits your documents into chunks, embeds each chunk as a vector, stores those vectors in an index, and at query time retrieves the most relevant chunks and pastes them into the model’s prompt as context. It exists because a model has a fixed knowledge cutoff and cannot see your private corpus; retrieval closes both gaps without retraining.
When should you use RAG instead of fine-tuning?: Use RAG when the problem is knowledge — facts that change, that are private to you, or that must be cited. Retrieval lets you add, update, or delete a document by re-indexing it in seconds, and it lets the answer point at the source. Use fine-tuning when the problem is behavior — a tone, a format, a skill, or a domain style the model should internalize. The two are not rivals: many production systems fine-tune for form and retrieve for facts. The honest rule is that fine-tuning teaches the model how to say things, and RAG gives it something true to say.
Why is chunking so important in a RAG system?: Because retrieval can only return whole chunks, so the chunk boundary decides whether an answer is retrievable at all. Chunks that are too small split a fact across a boundary — no single chunk contains the whole answer, so retrieval returns a fragment. Chunks that are too large bury the relevant sentence in unrelated text, which dilutes the embedding and drops the chunk’s match score, so it loses to sharper chunks. There is a sweet spot in between, and adding overlap between adjacent chunks buys back facts that straddle a boundary. It is the cheapest lever with the biggest effect on retrieval quality, which is why it is worth tuning against real questions rather than guessing.
What is reranking and why add a second retrieval stage?: Reranking is a second, slower, more accurate pass over the candidates the first stage retrieved. The first stage — a vector index — is fast because it compares a query vector to millions of document vectors independently, but that independence is exactly why it is imprecise. A reranker is a cross-encoder that reads the query and one candidate together and scores their true relevance, which is far more accurate but far too expensive to run over the whole corpus. So you retrieve maybe 100 candidates cheaply, rerank them to pick the best 5, and feed only those to the model. The cost is latency: reranking N candidates adds N model calls, which is the trade you dial against your latency budget.
How do you evaluate a RAG system?: By measuring retrieval and generation separately, because they fail differently. Retrieval is scored with recall — did the chunk containing the answer make it into the retrieved set? — against a golden set of questions with known answers. Generation is scored on faithfulness (also called groundedness): is every claim in the answer actually supported by the retrieved context, or did the model make something up? A common framing is the RAG triad: context relevance (are the retrieved chunks relevant to the question), groundedness (is the answer supported by those chunks), and answer relevance (does the answer actually address the question). A high final answer with low retrieval recall is luck, not a working system, so you watch both.

Design a RAG System

Questions

Related explainers