8.7 Retrieval-augmented generation

What this is and why it exists

Retrieval-augmented generation grounds a model in documents you control: find the relevant material, put it in the request, and require the answer to come from it. It is the standard architecture for answering questions over private data, and it works. The mistake that makes it hard to debug is measuring only the final answer — because when an answer is wrong you have to know whether retrieval missed the passage or generation ignored it, and those need entirely different fixes.

The vocabulary

• Chunk — one retrievable piece of a document.
• Overlap — text repeated between adjacent chunks so an idea is not cut in half.
• Retriever — the component that finds candidate chunks.
• Reranker — a more careful model that reorders the candidates.
• Context assembly — choosing what actually goes into the request.
• Grounding — requiring the answer to come from the retrieved material.
• Citation — pointing at which chunk supports which part of the answer.
• Recall at k — the share of questions whose answer is somewhere in the top k chunks.

The mental model

Five stages, and each can be improved independently once you can see which one failed. Chunk the documents. Retrieve candidates for the question. Rerank them. Assemble what fits into the request. Generate an answer that cites what it used.

Chunking is the least glamorous and most consequential decision, because a chunk is the unit of retrieval — anything that gets separated across two chunks can never be retrieved together, and no later stage recovers it.

Chunk too small and each piece lacks the context to be understood, and a passage about "it" has lost what "it" was. Chunk too large and the vector describes an average of several topics, which matches everything weakly and nothing well, and you spend context on material the question never needed. The workable range for prose is a few hundred tokens, and the number matters less than the rule: split on structure, not on character count. Documents have headings, sections, list items and rows; those boundaries are where meaning changes, and splitting there gives chunks that are about one thing. A fixed-width splitter cuts sentences in half and separates a table from its header, and that alone accounts for a great many disappointing systems.

Overlap exists for the boundary case: repeat a sentence or two between adjacent chunks so an idea spanning a boundary appears whole in at least one. It costs storage and a little duplication in results, and it is worth it.

Two additions that pay for themselves. Keep the document title and the section heading with every chunk, prefixed into the text that gets embedded — a chunk that says only "the limit is thirty days" is unretrievable, and one that says "Refunds — Time limits — the limit is thirty days" is found immediately. And store the source and position as metadata, because that is what makes citation possible later.

Retrieval, then reranking, is a deliberate two-stage design. The retriever is fast and imprecise, and it should be generous: fetch more candidates than you intend to use, because a chunk not retrieved is unrecoverable while an extra candidate is merely reordered away. Combine keyword and vector scoring here, for the reason the previous topic gave.

The reranker then reads the question and each candidate together and scores the match properly. That is far more accurate than comparing two independently produced vectors, and it is far too slow to run over the whole collection — which is exactly why it runs over a few dozen candidates instead of millions. This two-stage arrangement is the single most reliable quality improvement in the topic, and the reranker is a small encoder of the kind the language module described, which is why it is cheap.

Context assembly is where the placement rules from the token topic apply. Order by relevance so the strongest evidence sits at an edge rather than buried in the middle. Deduplicate — near-identical chunks from a document and its copy waste the budget and add nothing. Include the source label with each chunk so the model can cite it. And stop well short of the window: more retrieved material past a point makes answers worse, because weak passages dilute strong ones and the middle is attended to poorly.

Grounding and citation are what make the output checkable. Instruct the model to answer only from the provided material and to say so when the material does not contain the answer — the positive-rule form from the prompting topic. Require each statement to name the chunk supporting it, and then verify the citations in code: check the cited identifier exists among what you actually sent. That one check catches invented references immediately and costs nothing, and it converts citation from a stylistic flourish into an enforced property.

Without it, citation is decoration. A model asked to cite will produce citation-shaped text whether or not it used the sources, and a reader seeing references assumes they were checked. If you display citations, verify them.

Then the measurement point, which is the topic's real content. Measure retrieval and generation separately, from the first day.

Retrieval quality, measured with a set of questions each paired with the chunk that answers it: what share of questions have the right chunk in the top k? That single number tells you the ceiling, because if the passage was never retrieved, no amount of generation quality can rescue the answer. Build that set early — fifty questions from real use, each with its answering passage identified — and it will pay for itself many times over.

Generation quality, measured by giving the model the correct passages and asking whether the answer is right, complete and grounded in them. That isolates the model's behaviour from the retriever's.

With both numbers, every failure has an address. Retrieval low: fix chunking, add hybrid search, add a reranker. Retrieval high and generation low: fix the prompt, the assembly order, or the amount of context. Both high and users still unhappy: the questions being asked are not the questions you tested, which is its own useful finding. Conflating the two is why so many of these systems are debugged in the wrong place, and it is entirely avoidable by building the two measurements before building the third improvement.

What you should now be able to explain or do

Name the five stages and say what each contributes. Chunk on structure rather than width, choose overlap, and prefix titles and headings. Store source metadata and say what it enables. Design retrieval and reranking as two stages and say why the reranker is more accurate and affordable. Assemble context with ordering, deduplication and a deliberate stop short of the window. Enforce grounding with positive instructions and verify citations in code. Build separate retrieval and generation measurements and use them to locate a failure.

Check yourself