13.4 Capstone 4 — an agent with tools and evals

What this is and why it exists

The final capstone is an agent that completes a real multi-step task, with tools that fail gracefully, an enforced cost ceiling and a documented failure taxonomy. It proves you can engineer autonomy rather than prompt it. The decision that makes or breaks it comes first and is the hardest part: choose a task with a success criterion a program can check, because "it usually works" is not an evaluation and an agent you cannot score is an agent you cannot improve.

The vocabulary

• Multi-step task — one where the steps depend on what earlier steps found.
• Checkable criterion — a success condition a program can evaluate.
• Tool contract — the described function and argument shape.
• Trace — the recorded structure of a run.
• Cost per task — total spend for one completed task.
• Step ceiling — the enforced maximum number of iterations.
• Trajectory evaluation — judging the path as well as the outcome.
• Failure taxonomy — the ways it goes wrong, grouped by cause.

The mental model

Start by earning the agent, because the first agent topic's test applies to your own project: write down the steps on a typical task, and if you can write them down, you have described a chain. Choose a task where the steps genuinely depend on what is found — the search result decides whether a second search is needed, the error decides which repair applies, the document type decides the extraction.

Then the criterion, which is the harder half. A program must be able to check the result. Good shapes: the produced code passes tests you wrote; the extracted values match a reference; the file exists with the right structure; the answer matches a known result; the requested change appears in a system you can query. Poor shapes: anything requiring a person to judge quality, anything where "it looks right" is the only test.

That constraint narrows the task space considerably and it is worth accepting, because everything else in this capstone depends on being able to score a run automatically. Tasks that work well: fixing a failing test in a small repository; extracting structured records from a set of documents and validating them; researching a question across several sources and producing a cited answer checked against a reference; carrying out a multi-step change and verifying the end state.

Build the loop yourself, at least once, before reaching for a framework. Forty lines and you know what is underneath: the scaffold, the state, and the termination logic. Then use a framework for persistence and resumability if the task needs them — and be able to say in your write-up what it is doing for you, which is a question that gets asked.

Tool design is where most of the quality is, and tool errors are the normal case rather than the exception.

Write each description as a prompt: what it does, when to use it, when not to, what each argument means with units and formats, and what it returns. Keep the set small — a handful used correctly beats fifteen used approximately. And make errors actionable: "no record with that id — the format is eight digits" leads to a correction, while a bare failure code leads to a retry loop.

Handle failure deliberately, because it is what the reviewer is looking at. Every tool should return a structured failure rather than raising; the loop should decide per tool whether to retry, and reads are safe to retry while non-idempotent writes are not; and anything irreversible in your task should sit behind a confirmation, even if you are the only user, because the design is what is being assessed.

Tracing, cost and step ceilings are the three that make an agent safe to leave running, and they are also the three that most portfolio agents lack.

Trace every run: each model request with its full prompt, each response, each tool call with arguments and result, nested so a step sits inside what caused it, with timing and token counts. Use an existing tracing tool or write it to files — the mechanism matters less than having the record. Without traces you cannot analyse anything, and the analysis is what elevates this project.

Measure cost per task and report the distribution rather than the mean, because a small share of tasks running to the limit is the normal shape and those are most of the spending.

Enforce a step ceiling and a token budget, checked every iteration, in code. Add loop detection — the same tool with the same arguments twice, the same error repeating, no new information across several steps — and record which pattern fired.

Then trajectory evaluation and the failure taxonomy, which are what elevate this above a chatbot demonstration.

Report three numbers, not one: success rate, steps to completion and cost per task. An agent that succeeds equally often at twice the cost in three times the steps got worse, and the single number hides it.

Then judge the paths. Take twenty runs and ask of each: did it choose the right tools, in a sensible order, without repeating itself, and did it stop when it had enough? A run that reached the right answer in fifteen steps with three abandoned tools is a success with no margin, and the next slightly harder task will exhaust the budget.

And write the failure taxonomy down, grouped by cause with a count for each. Typical groups: chose the wrong tool; called the right tool with wrong arguments; misread a tool result; looped without progress; gave up early; produced a plausible but unverified answer; hit the budget. A documented failure taxonomy is more impressive to an interviewer than a success rate, because it demonstrates that you looked at what happened rather than only at whether it worked — and it is the thing a candidate who has actually operated an agent can produce and a candidate who has not cannot.

The write-up should show the trace of one failure, in full, with your analysis of where it went wrong and what you changed. That single artefact carries more evidence of capability than the success rate does, and it is the closing move of the project: anybody can show a run that worked; showing that you understood one that did not is the whole point of this capstone.

What you should now be able to explain or do

Apply the write-down-the-steps test to your own task and earn the agent. Choose a task with a program-checkable criterion and give examples of good and poor shapes. Build the bare loop before adopting a framework, and say what the framework does for you. Write tool contracts as prompts with actionable errors and deliberate failure handling. Trace every run with nesting, timing and tokens. Report success rate, steps and cost as distributions. Judge trajectories with the four questions. Write a failure taxonomy with counts, and present one failure trace with your analysis.

Check yourself