8.5 Structured output and tool schemas

What this is and why it exists

Structured output is what makes a language model composable with ordinary code. Free text has to be parsed by something that guesses; a validated object can be passed to a function. This topic is the layers that make that dependable — a declared schema, constrained generation, validation with a repair loop — and one rule that is not negotiable: never feed unvalidated model output into anything that acts.

The vocabulary

• Schema — a declarative description of the structure data must have.
• JSON mode — a setting asking for syntactically valid JSON.
• Schema-constrained output — generation that cannot violate a supplied schema.
• Tool calling — describing functions the model may request, and the shape of their arguments.
• Validation — checking a returned object against a declared model.
• Repair loop — feeding the validation error back and asking again.
• Refusal path — the defined behaviour when a request cannot be satisfied.

The mental model

Start from what a schema is for. JSON Schema describes itself as "a declarative format for 'describing the structure of other data'", and that is the point: the shape lives in one place, as data, rather than being implied by a paragraph of prose in a prompt and re-implied by parsing code somewhere else. Write it once, use it to constrain generation, to validate the result, and to document the contract.

Then the distinction this topic exists for. "JSON mode" asks for syntactically valid JSON, and syntactically valid JSON can be completely wrong for your purposes: a missing required field, a string where a number belongs, a category outside your enumeration, a date in a format you cannot parse, an extra field your code ignores while the value you needed is absent. Valid JSON is not valid data, and treating the two as the same is how a pipeline breaks two weeks after it appeared to work.

Schema-constrained generation is the stronger tool, and it is the decoding topic applied here: mask every token that would break conformance, and the output cannot violate the schema, because no invalid token was available at any step. Where the interface offers it, use it — it removes syntax and structure failures entirely and makes the retry loop rare rather than routine. It still constrains shape rather than truth: a perfectly conforming object can contain a fabricated value, and nothing about the schema says otherwise.

Tool calling is the same contract facing the other way. Instead of describing the shape of an answer, you describe the functions the model may request and the shape of their arguments, and it responds with a name and an argument object rather than prose. That is what lets a model act rather than only answer, and everything above applies with the stakes raised, since these arguments reach code that does something.

Three rules for tool contracts. Describe each function as if for a colleague who cannot see your code — what it does, when to use it, what each argument means, what the units are; vague descriptions are the main cause of wrongly chosen tools. Keep the argument shapes narrow: enumerations rather than free strings wherever the set is known, because an enumeration is checkable and a free string is a guess. And validate the arguments before executing, always — a returned argument object is a request, not an authorisation, and the checks you would apply to input from a web form apply here identically.

Validation with a repair loop is the layer underneath all of it. Declare your model — Pydantic describes a model as a class inheriting from BaseModel that defines "fields as annotated attributes" — and validate the returned data against it. The guarantee is worth stating exactly: "untrusted data can be passed to a model and, after parsing and validation, Pydantic guarantees that the fields of the resultant model instance will conform to the field types defined on the model." When it does not conform, "Pydantic will raise a ValidationError exception whenever it finds an error in the data it's validating", and usefully, "a single exception will be raised regardless of the number of errors found, and that validation error will contain information about all of the errors and how they happened."

That last detail is what makes the repair loop work: the error names every problem at once, so you can hand it back and ask for a corrected object in one round trip rather than discovering the faults one at a time.

The repair loop is a normal part of the design, not a workaround, and it needs three things to stay safe. A hard attempt limit — two retries, then fail — because an unbounded loop against a paid interface is a bill and a hang. The validation error passed back verbatim, since a specific message about which field failed is far more useful to the model than a request to try again. And a defined outcome when the limit is reached: log the raw output, return a failure your caller can handle, and never fall back to a partially parsed object, which is how malformed data enters a system quietly.

Then the refusal path, which is a design decision made in advance. A system that declines a request it cannot satisfy is safer than one that produces something plausible, and plausible is the default behaviour unless you build otherwise. So put refusal into the contract as a first-class outcome: a schema in which the response is either a result or a stated inability, with a reason. Show it in your examples. Test it — the failure taxonomy from the prompting topic is where those cases come from. A schema with no way to say "I cannot" is a schema that forces a guess, and you will get one.

The rule that closes the topic. Model output is untrusted input. Validate before use, and never let it reach anything that acts — a query, a command, a file path, a request to another service, a payment — without the same checking you would apply to a form filled in by a stranger. The model is not adversarial; the person who wrote the document it read may be, which is the injection problem the safety topic develops.

What you should now be able to explain or do

Say what a schema is for and why the shape belongs in one place. Distinguish syntactically valid JSON from valid data and give three ways the first fails you. Explain what schema-constrained generation guarantees, and what it does not. Write a tool contract with clear descriptions and narrow argument shapes, and validate arguments before executing. Validate with a declared model, quoting what the guarantee actually is. Build a repair loop with an attempt limit, the error passed back, and a defined failure. Design a refusal path into the schema. State the rule about unvalidated output.

Check yourself