8.8 Advanced RAG

What this is and why it exists

A grounded system that returns plausible nonsense is the characteristic failure of this architecture, and the techniques in this topic are the repair kit. Each one fixes a specific failure — not retrieval in general. That distinction is the whole lesson: stacking every technique because a system is disappointing is expensive superstition, and the diagnosis takes an afternoon and tells you which single fix applies.

The vocabulary

• Keyword scoring — ranking by term frequency against how rare each term is.
• Hybrid search — combining a keyword score and a vector score into one ranking.
• Rank fusion — merging ranked lists by position rather than by raw score.
• Query rewriting — reformulating the user's question before searching.
• Hypothetical answer search — generating a plausible answer and searching with it.
• Multi-query — issuing several reformulations and pooling the results.
• Graph retrieval — retrieving over entities and relationships rather than loose text.
• Contextual retrieval — attaching surrounding context to each chunk before storing it.

The mental model

Diagnose first, and the diagnosis is three questions asked in order.

Is the query the problem? Take the failing questions and search manually with a better-phrased version. If the right passage appears, retrieval is fine and the question was poorly formed for search — a pronoun with no antecedent, a follow-up that depends on the previous turn, an acronym, a phrasing nothing in your corpus uses. That is a query problem.

Are the chunks the problem? Look at what the failing questions should have matched. If the answering text is split across two chunks, or sits in a chunk about six other things, or lost the heading that said what it was about, no ranking improvement will help. That is a chunk problem.

Is the ranking the problem? If the right chunk is retrieved but sits at position forty and you use the top five, the material exists and the ordering failed. That is a ranking problem.

Each has its own fix, and applying the wrong one costs money and changes nothing.

Hybrid search is usually the first improvement worth making, and it fixes the specific weakness of vector search: exact terms. Classical keyword scoring ranks by how often a term appears in a document against how rare that term is across the collection, so it excels precisely where embeddings are weak — product codes, error numbers, proper nouns, and any term the embedding model never saw. Vector search excels on paraphrase, where the words differ and the meaning does not. Running both and merging catches both cases.

Merge by rank rather than by raw score, since the two scores are on incomparable scales and normalising them is fragile; combining by position in each list is simple and robust. If your system fails on identifiers and succeeds on descriptions, this is your fix and you can stop here.

Query rewriting fixes questions that are badly formed for search. The commonest case is conversational: "what about the second one?" carries no searchable content at all, and rewriting it against the conversation history into a standalone question is a small model call that repairs a whole class of failure. Other cases are expanding acronyms, adding synonyms your corpus actually uses, and splitting a compound question into parts.

Hypothetical answer search is a neat variation for a specific mismatch: questions and documents are written differently — a question is short and interrogative, a passage is long and declarative — so a question's vector sits somewhere unlike any passage's. Generate a plausible answer to the question, embed that instead, and search with it: an answer-shaped vector matches answer-shaped text. It costs one extra model call, and the generated answer may be factually wrong without mattering, since it is only ever used to search.

Multi-query issues several reformulations at once and pools the results, which raises recall at the cost of several searches and a larger candidate set to rerank. Use it where missing a passage is expensive and latency is not critical.

Graph retrieval is the heavier tool and it answers a different question. Extract entities and their relationships into a graph, and retrieve by traversing it. This handles questions that require joining facts across documents — who reports to the person who approved this, which components are affected by a change to that one, what connects these two things — where no single chunk contains the answer and so no amount of chunk retrieval can produce it.

That is its whole justification, and it should be the reason you reach for it. The cost is substantial: entity extraction over the corpus, a schema, a graph to maintain, and re-extraction when documents change. If your failing questions are answerable from one passage each, this is a large amount of work for nothing.

Contextual retrieval and late chunking address the same weakness in naive chunking, from different directions. Contextual retrieval prefixes each chunk with a short generated description of where it sits and what it refers to before embedding it, so a chunk beginning "this applies only to enterprise accounts" carries what "this" was. It costs one model call per chunk at indexing time — paid once — and it is one of the highest-value improvements available, because it repairs the pronoun-and-antecedent problem that ordinary chunking creates by construction.

Late chunking inverts the order: embed the whole document with a long-context model so every token's representation is informed by the full document, then cut the token representations into chunks. Each chunk's vector then reflects the whole document's context without any generation step. Both fix the same defect; the first is more widely supported and the second is cheaper where it is available.

The rule this topic exists for. Stacking techniques without a diagnosis produces a system that is slower, more expensive, harder to reason about, and no better — and worse, one where you cannot tell which part is helping. Diagnose, apply the one fix that matches, measure the retrieval number from the previous topic, and only then consider the next. If a technique does not move that number, remove it. Everything else is expensive superstition.

What you should now be able to explain or do

Run the three diagnostic questions and classify a failure as query, chunk or ranking. Explain why hybrid search fixes exact-term failures and why merging by rank beats merging by score. Apply query rewriting to conversational and acronym failures. Say what hypothetical answer search corrects and why a wrong generated answer is harmless. Say what multi-query buys and costs. State the one justification for graph retrieval and its maintenance cost. Explain contextual retrieval and late chunking as two fixes for the same defect. Measure after each change and remove what does not help.

Check yourself