9.11 Cost control and production failure modes

What this is and why it exists

An agent is a loop that spends money on every iteration, and the failure mode is not that it stops — it is that it does not. Averages look fine while the tail runs away, and one pathological task or one determined user turns a working feature into an invoice you will remember. This topic is the financial governors that make that impossible rather than unlikely, and the degradation path that decides what a person sees when a limit is reached.

The vocabulary

• Token budget — a cap on tokens for a scope, enforced rather than intended.
• Step ceiling — a maximum number of iterations, applied regardless of state.
• Loop detection — noticing that the agent is repeating itself.
• Model routing — sending different steps to different models by difficulty.
• Prompt caching — reusing the processed form of a repeated prefix.
• Semantic caching — reusing a previous answer for a sufficiently similar request.
• Degradation path — what happens when a limit is reached.
• Tail — the small share of tasks costing many times the median.

The mental model

Understand the shape of agent spending before controlling it. Cost per task is not the number of steps times the cost of a call, because the transcript grows: step twenty carries everything from steps one to nineteen, so the later steps cost several times the early ones. A task that takes twice as many steps costs considerably more than twice as much.

And the distribution is heavily skewed. Most tasks finish quickly; a small share go badly and run to the limit, and those tasks are most of the bill. This is why an average is reassuring and misleading, and why every control below is about the tail rather than the mean.

The budgets, in order of how much they save.

A per-task token budget, checked every iteration and enforced in your code before the request goes out. This is the single most important control, because it bounds the worst case absolutely: whatever happens — a loop, a pathological input, a tool returning enormous results — the task cannot cost more than the number you chose. Enforced, not intended: a limit in the prompt is a suggestion, and a limit in a variable that nothing reads is a comment.

A per-user budget over a period, because per-task limits still allow one user to run a thousand tasks. Set it from what legitimate heavy use looks like, and return a clear message rather than failing obscurely when it is reached.

A step ceiling regardless of budget, since a loop of cheap steps is still a loop and still latency.

A maximum output length on every call, because the default is generous and unbounded generation is unbounded cost.

A tool-result size limit, which people forget: a tool returning a large payload puts it into the transcript, where it is paid for on every subsequent step. Truncate at the tool boundary, with a note saying it was truncated.

Loop detection is not exotic — loops are the normal failure mode. An agent that cannot make progress will try again, and again, plausibly each time. Detect it by comparing recent steps: the same tool with the same arguments twice in a row, the same error returned repeatedly, two tools alternating, or several steps adding nothing new to the transcript. Stop when a pattern matches, and record which pattern fired, because that record is your failure taxonomy for the next improvement.

Routing is frequently the largest saving available, and it is under-used. Not every step needs the strongest model. Deciding which of four tools to call, extracting a field from a result, summarising a page, checking whether output is on-topic — these are decisions a small, fast, cheap model makes as well as a large one, while the hard reasoning and the final answer stay with the capable model. Routing steps by difficulty routinely cuts the cost of a task by a large fraction with no change in what the agent can do.

Build it as a policy rather than as scattered choices: a table saying which model handles which step type, so it is visible, changeable and measurable. Then verify it with your evaluation — route a step down, measure the success rate, and keep the change only if it holds. That is what separates routing from guessing.

Caching comes in two forms and both are worth having. Prompt caching reuses the processed form of a repeated prefix, which is exactly the shape of an agent: the same system prompt and tool descriptions on every call. Where the interface offers it, arranging your requests so the stable part comes first is a small change for a real saving. Semantic caching returns a previous answer for a sufficiently similar request, which suits read-only lookups and suits nothing where the answer depends on time or on who is asking. Cache the tool results too — the same search repeated within a task should not be paid for twice, and an agent will repeat it.

Then the degradation path, which is a design decision made in advance.

When a limit is reached, stopping silently is the worst option, and it is the default if you do not choose. Rank the alternatives: return the partial work with a clear statement of what was completed and what was not; explain what would let it continue — more budget, a narrower request, a person's input; hand off to a person with the transcript, so the work is not lost; or fail cleanly with something the caller can act on. Any of those beats silence.

Two details make it usable. Say which limit was hit, because "budget exhausted after eighteen steps" tells a user something and "unable to complete" does not. And preserve the partial work where it has value — a half-finished analysis with three of five sections is worth more than nothing, and throwing it away because the task did not complete is a choice you made by not deciding.

And the operational habits that make the whole thing manageable. Log tokens and cost per task with the user and the task type attached, from the first day, or you cannot answer where the money went. Alert on the tail rather than the average — the ninety-fifth percentile moving is the early warning; the mean moving is the bill arriving. And review the most expensive tasks weekly, because they are a short list and they tell you exactly what to fix.

What you should now be able to explain or do

Explain why cost grows faster than step count and why the distribution is skewed. Implement per-task and per-user budgets, a step ceiling, an output cap and a tool-result limit, enforced in code. Detect loops by comparing recent steps and record which pattern fired. Build a routing policy by step type and verify it against your evaluation. Use prompt caching, semantic caching and tool-result caching where each fits. Choose a degradation path in advance, naming the limit and preserving partial work. Log per-task cost and alert on the tail.

Check yourself