9.9 Evaluating and observing agents
You can measure whether an agent is getting better.
Before:08. Large Language Models
Agent evaluation looks at trajectories, not just outcomes: where runs go wrong, how many steps and how many rupees a task costs, with tracing tools as the observability layer and recorded runs replayed as regression suites. It sits near the module's end because it measures everything built. The single-number trap is success rate alone — an agent that succeeds at triple the cost, or by taking an unsafe route, did not simply succeed.
Work through these
Trajectory evaluation vs. outcome evaluation
Judging the path an agent took against judging only whether it reached the right answer. Both matter, and an agent that stumbles to a correct answer is not reliable.
Tracing: LangSmith, Langfuse, OpenTelemetry
Recording every model call, tool call and decision in a run, using either a purpose-built service or the open standard for it. Without traces, agent debugging is guesswork.
Task success rate, steps-to-completion, cost-per-task
Three numbers that between them describe agent quality: how often it succeeds, how much work it takes, and what each task costs. Reporting only the first hides the other two.
Replay and regression testing on recorded runs
Saving real runs and replaying them after a change is how you find out whether the change helped. It is the agent equivalent of a regression suite.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.