13.3 Capstone 3 — RAG over a real corpus

What this is and why it exists

The third capstone is a grounded question-answering system over a real corpus, with citations, a refusal path and an evaluation suite. The evaluation suite is the differentiator: almost everybody can assemble a retrieval pipeline from a tutorial, and almost nobody can show a measured retrieval number and a regression run. The trap is skipping refusal — a system that answers confidently when retrieval found nothing is a liability demonstration, and any reviewer who knows this architecture will probe exactly that.

The vocabulary

• Corpus — the document collection the system answers from.
• Ingestion — getting documents in and into a searchable form.
• Chunk — the retrievable unit.
• Reranking — reordering retrieved candidates with a more careful model.
• Citation enforcement — verifying in code that cited sources exist.
• Refusal on no evidence — declining when nothing relevant was retrieved.
• Golden set — fixed questions with known good answers.
• Regression run — that set, executed on every change.

The mental model

Choose a corpus you can judge, which is the equivalent of the first capstone's decision. You must be able to tell a correct answer from a plausible one without a second opinion, or you cannot evaluate anything. Good choices: documentation for a tool you use, a body of public regulation or policy, papers in an area you know, your own accumulated notes, or the manuals for something you own. A corpus of a few hundred documents is plenty; a very large one adds infrastructure without adding evidence of skill.

Ingestion is unglamorous and it is where projects lose weeks, so plan for it. Extracting text from documents is uneven — tables become nonsense, columns interleave, headers and footers repeat on every page, scanned pages need character recognition. Look at the extracted text before building anything on it, because a retrieval system over garbled text fails in ways that look like retrieval failures and are not.

Keep the metadata as you ingest: source, title, section, page, date. It is what makes citation possible and filtering meaningful, and adding it afterwards means reprocessing everything.

Chunking quietly determines the ceiling on everything that follows, exactly as the retrieval topic said, so treat it as a decision rather than a default. Split on structure — headings, sections, list items — rather than on a character count. Overlap a little. Prefix the document title and section heading into every chunk's embedded text, which is the single cheapest improvement available and turns unretrievable fragments into findable ones. And read fifty chunks, because that is how you discover a splitter cutting tables in half.

Retrieval plus reranking is the architecture, and measuring retrieval separately is what makes it debuggable.

Retrieve generously with hybrid search — keyword scoring and vector similarity merged by rank — then rerank the candidates with a cross-encoder over question and passage. Two stages, for the reasons the retrieval topic gave.

Then measure retrieval on its own, which is what almost no portfolio project does. Build a set of questions each paired with the chunk that answers it — fifty is enough — and report what share have the right chunk in the top k. That single number is the ceiling on your system, because a passage never retrieved cannot be used, and being able to state it is worth more in a conversation than any architectural choice you made.

Report it before and after each improvement. "Adding hybrid search moved recall at five from 0.72 to 0.88" is the sentence that separates this project from a tutorial.

Citation enforcement is a code check, not a prompt. Ask the model to cite the chunk supporting each statement, and then verify in code that every cited identifier is among the chunks you actually sent. Refuse or flag the answer when it is not.

This costs a few lines and it converts citation from decoration into an enforced property. A model asked to cite will produce citation-shaped text whether or not it used the sources, and a reader seeing references assumes somebody checked. If you display citations, verify them — and say in your write-up that you do, because reviewers who know this architecture will ask.

Refusal on no evidence is the part reviewers probe and the part most projects skip.

Build it deliberately. Threshold on retrieval: if the best retrieved chunk scores below a level you chose by inspection, do not call the model at all — answer that the corpus does not cover it. Make refusal a first-class outcome in the response schema, so the system returns either an answer with citations or a stated inability with a reason. Show it in your examples, so refusing is a behaviour the model has seen. And test it: put questions in your golden set that the corpus genuinely cannot answer, and assert that the system declines.

That last one is the difference. A system that answers confidently when retrieval found nothing is a liability demonstration, and the question a knowledgeable reviewer asks is "what happens if I ask about something not in there" — which you should be able to answer by inviting them to try it.

The golden set with regression runs is the differentiator, and it is straightforward. Fifty to a hundred questions with agreed-good answers, covering the common cases, the hard multi-document ones, the ambiguous ones, and the ones that must be refused. Version it in the repository. Run it on every change — a different chunk size, a new embedding model, an edited prompt, a reranker added — and record the numbers.

Assert on properties rather than exact strings: does it contain the required fact, does it cite something real, does it refuse where it should, does it stay within length. And report retrieval and generation separately, so a regression has an address.

The write-up is what turns the work into evidence. Show the retrieval number and how it moved with each change. Show the refusal behaviour, including an example. Show a regression run. State what the corpus does not cover and what the system therefore cannot answer. Most candidates cannot show any of that, and showing it is the point of building this capstone rather than a chatbot.

What you should now be able to explain or do

Choose a corpus you can judge and say why that is the deciding property. Ingest with metadata, and inspect extracted text before building on it. Chunk on structure with heading prefixes, and read a sample. Build hybrid retrieval with reranking. Measure retrieval separately with a question-and-passage set, and report it before and after each change. Enforce citations in code. Build refusal as a retrieval threshold, a schema outcome, an example and a test. Maintain a golden set with property assertions and separate reporting.

Check yourself