Every LLM feature that feeds a program instead of a person needs structured output. Extraction, classification, routing, tool arguments, agent handoffs, eval judges: all of them break the moment the model returns a sentence where your code expected an object.
For a long time the fix was a prompt that said "respond only with JSON" and a regex that stripped the markdown fences. That works most of the time, and "most of the time" is the problem. At 100,000 calls a day, a 0.5% parse failure rate is 500 broken requests.
Today every major API and every serious open-source inference server can guarantee that output matches a JSON schema. That solves syntax. It does not solve correctness, and teams that treat the two as the same thing ship confident, perfectly formatted, wrong data. This guide covers both halves.
Key Takeaways
- There are four approaches: prompting, JSON mode, schema-constrained output, and strict tool calling. Use constrained output for anything a program consumes.
- Constrained decoding guarantees the output parses and matches the schema. It does not guarantee the values are true.
- Schema design is prompt design. Field names, descriptions, enums and field order all change what the model writes.
- Put a reasoning field before the answer fields when the task needs thought. Order is generation order.
- Still validate in code. Refusals,
max_tokenstruncation and unsupported schema keywords all produce output your schema did not promise.- Measure semantic accuracy per field, not parse rate. Parse rate is 100% by construction and tells you nothing.
The Four Ways to Get Structure
| Approach | How it works | Syntax guarantee | Schema guarantee | Use it for |
|---|---|---|---|---|
| Prompt + parse | Instructions and examples, then JSON.parse |
No | No | Prototypes, models without support |
| JSON mode | Decoder forced to emit valid JSON | Yes | No | Legacy code paths |
| Schema-constrained output | Decoder forced to follow a JSON Schema | Yes | Yes (supported subset) | Extraction, classification, judges |
| Strict tool calling | Tool arguments constrained to the tool's schema | Yes | Yes (supported subset) | Agents choosing among actions |
Prompt + parse is where everyone starts. It fails in predictable ways: markdown fences, a friendly preamble ("Here is the JSON:"), trailing commas, unescaped quotes inside strings, and truncated objects when the model runs out of tokens. You can patch each one, and you'll find a new one next week.
JSON mode was the first API-level fix. The output is guaranteed to be valid JSON, but not your JSON. Fields can be missing, renamed, or nested differently. It's mostly superseded now.
Schema-constrained output is the current default. You pass a JSON Schema, and the provider guarantees the response conforms. OpenAI (strict json_schema response format), Anthropic (output_config.format), and Google Gemini (response schema) all offer it, and open-source servers such as vLLM and SGLang support it through grammar backends like XGrammar, Outlines and llguidance.
Strict tool calling applies the same guarantee to tool arguments. If the model is choosing which action to take as well as filling in parameters, this is the right shape. If there's only one possible "tool" and it exists just to get JSON back, use schema-constrained output instead; it's simpler and avoids tool-selection failures.
How Constrained Decoding Works
An LLM generates one token at a time by sampling from a probability distribution over its whole vocabulary. Constrained decoding adds one step: before sampling, it masks out every token that would make the output invalid under a grammar, then samples from what's left.
The schema is compiled into a grammar (usually a pushdown automaton, since JSON nests). At each step the engine knows the current parse state. Just after {"status": with status defined as an enum of "open" and "closed", the only legal tokens are ones that begin "open" or "closed". Everything else gets probability zero.
Three consequences matter in practice:
- The guarantee is real. The output cannot violate the grammar, because invalid tokens are never sampled. Parse rate becomes 100% for completed responses.
- There is a compile cost. Turning a schema into a grammar and precomputing token masks takes time. Providers cache compiled grammars, so the first request with a new schema is slower and later ones are not. Anthropic, for example, caches a compiled schema for 24 hours. Don't generate schemas dynamically per request if you can avoid it.
- The model can be pushed off its preferred path. If the model "wanted" to write something the grammar forbids, masking forces the next-best legal token. When your schema fights the model's natural output, quality drops even though the format is perfect. This is why schema design matters so much.
Supported schema subsets
No provider supports all of JSON Schema. Recursive schemas, numeric bounds (minimum, maximum), string length limits and complex array constraints are common gaps, and most providers require additionalProperties: false on every object. The exact list differs by provider and changes over time, so check the current docs.
The practical rule: keep the schema you send to the model simple, and enforce the rest in code. The official Anthropic Python and TypeScript SDKs do this automatically: they strip unsupported constraints before sending and validate them client-side after. If your SDK doesn't, do it yourself.
A Minimal Production Example
Here's a support-ticket triage call using Claude with a Zod schema. The pattern is identical with Pydantic in Python or with other providers' SDKs.
import Anthropic from "@anthropic-ai/sdk";
import { z } from "zod";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
const Triage = z.object({
reasoning: z.string().describe(
"Two or three sentences on what the customer needs and why you chose the category."
),
category: z.enum(["billing", "bug", "feature_request", "account_access", "other"]),
severity: z.enum(["low", "medium", "high", "critical"]),
product_area: z.string().nullable().describe(
"The product area named in the ticket, or null if none is named. Do not guess."
),
needs_human: z.boolean(),
});
const client = new Anthropic();
const response = await client.messages.parse({
model: "claude-opus-5-5",
max_tokens: 2000,
system: "You triage customer support tickets for a B2B SaaS product.",
messages: [{ role: "user", content: ticketText }],
output_config: { format: zodOutputFormat(Triage) },
});
if (response.stop_reason !== "end_turn" || !response.parsed_output) {
// refusal, max_tokens truncation, or a parse failure: route to fallback
throw new TriageError(response.stop_reason);
}
const triage = response.parsed_output; // fully typed
Notice what's doing the work beyond the format constraint: the reasoning field comes first, every enum is closed, product_area is explicitly nullable with an instruction not to guess, and the code checks stop_reason before trusting the result.
For tool calls, the equivalent is strict: true on the tool definition, which guarantees tool_use.input matches the tool's input_schema. Note that several current models (Claude Opus 5.5 among them) no longer accept forced tool choice. If you were forcing a single tool purely to get JSON back, schema-constrained output is the replacement.
Schema Design Is Prompt Design
The model reads your schema. Field names, descriptions and structure all influence the values it writes. A schema is a prompt that also happens to be a type.
Field order is generation order
Models generate left to right. If category comes before reasoning, the model commits to a category first and then writes a justification for a decision it already made. Put the thinking first and the answer after it.
{
"reasoning": "...",
"category": "...",
"confidence": "..."
}
This matters most for classification and judging. With models that have built-in reasoning (adaptive or extended thinking), the thinking happens before the structured output anyway, and a short reasoning field becomes less important. It's still useful as an audit trail: when a classification is wrong, the reasoning field tells you why.
Use closed enums, and give them an escape hatch
Free-text fields for categorical values produce "Billing", "billing issue" and "payments" for the same thing. Enums fix that. But an enum with no valid option for an input forces the model to pick a wrong one. Always include other or unknown, and track how often it's used. A rising other rate is a signal your taxonomy has drifted.
Make "not present" representable
The most common extraction failure is fabrication under schema pressure. If invoice_number is a required string, and the document has no invoice number, the model has to write some string. It will invent a plausible one.
Make absence a legal value: nullable, and a description that says "null if not stated in the document." Required-but-nullable is better than optional, because it forces the model to make an explicit decision for every field instead of silently skipping one.
Name fields for the model, not just for your code
amt is ambiguous. total_amount_including_tax is not. dt could be any date. invoice_issue_date is clear. If your internal model uses terse names, map them after parsing. The extra tokens in the schema are cheap; ambiguity is expensive.
Keep it shallow and small
Deeply nested schemas with dozens of fields degrade quality. The model has to track more structure and the grammar gets more complex. If an extraction has 40 fields across five sections, consider splitting it into a few focused calls that run in parallel. It's often more accurate and not much slower.
Does Constraining Format Hurt Quality?
Sometimes, and the research is mixed. A 2024 paper, "Let Me Speak Freely?" (Tam et al.), reported that strict format restrictions degraded performance on some reasoning tasks compared with free-form answers. Follow-up analysis, including a response from the Outlines team at .txt, argued that much of the gap came from prompt and parsing differences rather than from constrained decoding itself, and that well-designed structured prompts closed it.
What practitioners can take from this:
- For extraction and classification, constrained output is almost always a net win. The task is already structured.
- For tasks that need multi-step reasoning, let the model reason first, either through native thinking or a reasoning field placed before the answer. Don't force a bare answer into a tight schema.
- Don't assume, measure. Run your eval set with and without the constraint (parsing the unconstrained output as best you can) and compare accuracy, not just parse rate.
What Still Goes Wrong
A schema guarantee covers completed, normal responses. Production traffic includes the abnormal ones.
Truncation
If the model hits max_tokens mid-object, you get a partial, unparseable response. Constrained decoding can't fix this; it only controls which tokens appear, not how many. Check the stop reason, set max_tokens with generous headroom over your largest expected output, and alert on truncation rate. Long arrays (e.g. "extract every line item") are the usual cause.
Refusals
When a model declines a request for safety reasons, the response may not match your schema. On Claude this shows up as stop_reason: "refusal". Treat it as its own outcome with its own handling path, not as a parse error to retry.
Valid but wrong
This is the big one. The model returns {"severity": "low"} for an outage ticket. The JSON is perfect. The schema validator is happy. The value is wrong.
Defend against it with semantic validation in code, after the schema check:
- Cross-field rules.
line_itemsshould sum tosubtotal.end_dateshould be afterstart_date. Schemas can't express most of these; code can. - Grounding checks. For extraction, verify that extracted values actually appear in the source. An extracted invoice number that doesn't occur anywhere in the document is a hallucination. This one check catches a large share of extraction errors, and it's cheap.
- Range and format checks the provider's schema subset didn't support.
Retry with the error, not blindly
When semantic validation fails, a retry that includes the specific error usually fixes it:
Your previous answer failed validation:
- line_items total (1,240.00) does not match subtotal (1,420.00)
Re-read the document and correct the extraction.
Cap retries at one or two. If the second attempt fails, the input is genuinely ambiguous or the task is too hard for the model, and a human or a stronger model should see it. Libraries like Instructor wrap this validate-and-retry loop around Pydantic models if you don't want to write it yourself.
Streaming Structured Output
Users don't want to wait for a full object before seeing anything. Most SDKs stream structured output as partial JSON deltas, and parsers like partial-json or the SDKs' own helpers can turn incomplete JSON into a best-effort object as it arrives.
Two rules keep this safe:
- Render partial data, don't act on it. It's fine to show a summary field filling in. It's not fine to trigger a side effect from a half-parsed object. Run validation and side effects only on the final, complete response.
- Order fields for display. If the user-facing field comes first in the schema, it streams first. This conflicts with "reasoning first" for some tasks; pick based on whether latency or accuracy matters more for that call.
The same caution applies to tool arguments streamed incrementally: the partial input isn't validated until it's complete, so validate before executing the tool.
Schemas Are Contracts: Version Them
Once a schema feeds downstream code, a database or another agent, it's an API contract. Treat it like one.
- Keep schemas in code, in one place, and generate the JSON Schema from your Zod or Pydantic types. Don't hand-maintain both.
- Version them. Store a
schema_versionalongside every stored output so you know which contract a record was produced under. - Additive changes are safe; renames and enum removals are not. Removing an enum value breaks every stored record and every consumer that switches on it.
- Re-run evals on every schema change. Changing a description or reordering fields changes model behavior. It's a prompt change and deserves the same testing.
This matters even more in multi-agent systems, where free-text handoffs between agents are a reliability anti-pattern. Structured messages at every agent boundary make failures visible and testable. See Multi-Agent Orchestration Patterns for how to design those boundaries.
Measuring It
Parse rate is useless once you use constrained output; it's 100% by construction. Measure these instead:
| Metric | What it tells you |
|---|---|
| Field-level accuracy against a labeled set | Which specific fields the model gets wrong |
| Null rate per optional field | Whether the model is fabricating (too low) or giving up (too high) |
other / unknown enum rate |
Taxonomy drift |
| Semantic validation failure rate | Valid-but-wrong outputs caught by your checks |
| Retry rate and retry success rate | How often the first answer is wrong, and whether feedback helps |
| Truncation and refusal rate | Abnormal completions that bypass the schema guarantee |
| First-call latency after schema change | Grammar compile cost |
Track these per schema version, and alert on changes. A jump in null rate after a model upgrade is the kind of silent regression that never shows up as an error. Our guides on LLM metrics that actually matter and continuous eval pipelines cover how to wire these into monitoring.
Frequently Asked Questions
Should I still use JSON mode?
Only for models or endpoints that don't support schema-constrained output. JSON mode guarantees valid JSON but not your structure, so you still need full schema validation. Schema-constrained output is strictly better where available.
Structured output or tool calling?
Use structured output when you always want one shape of answer back. Use tool calling, with strict schemas, when the model must decide which action to take. Wrapping a single extraction in a fake tool was a workaround from before structured output existed, and it's no longer needed.
Do I still need Pydantic or Zod if the provider guarantees the schema?
Yes. They give you typed objects in code, generate the schema so it can't drift from your types, and enforce the constraints the provider's schema subset can't express. They're also where your semantic validation lives.
Does constrained decoding cost more?
The output tokens are the same or fewer, since there's no preamble or markdown wrapper. The main cost is first-request latency while a new schema compiles. Keep schemas stable so the compiled grammar stays cached. Prompt caching also helps, since schema-heavy requests often share a long fixed prefix; see Cutting LLM API Costs with Prompt Caching and Model Routing.
Can I use constrained decoding with self-hosted models?
Yes. vLLM and SGLang support JSON Schema, regex and grammar constraints through backends such as XGrammar, Outlines and llguidance. Grammar backends differ in supported features and in compile speed for large schemas, so benchmark with your real schemas before choosing.
How do I stop the model inventing values for missing fields?
Make absence representable (nullable fields with "null if not stated" descriptions), then verify extracted values against the source text in code. Track null rates per field over time so you notice if a model update changes the behavior.
The Bottom Line
Structured output used to be a parsing problem. Constrained decoding has solved that part: for supported schemas, malformed JSON is no longer something you need to handle.
What's left is the harder problem it used to hide. A schema guarantees shape, not truth. The teams that get reliable results treat the schema as a prompt (reasoning first, closed enums, explicit nulls, clear names), treat its output as untrusted input (semantic checks, grounding, targeted retries), and measure field-level accuracy instead of parse rate.
Get the format guarantee from the decoder. Get the correctness guarantee from your own code and your evals.
Related Posts
Latest Deployment news →Weekly Digest
Get the best AI engineering posts, weekly
No hype. Curated signal every Sunday.