Structured Output Contracts: Constrained Decoding, JSON Schema, and Why Your Agent Still Returns Garbage
Your agent passes schema validation but returns garbage. Learn why constrained decoding only enforces syntax—and how to build structured output contracts that work.
Why Your Agent Returns Garbage Even When the Schema Passes
Your JSON Schema validates. Your model complies. Your agent still ships malformed or semantically wrong payloads downstream.
That gap is the subject of this article. The core thesis is simple: a JSON Schema is a contract, not a guarantee of correctness. Constrained decoding enforces syntax, not meaning. When you hand a schema to a model and enable strict structured output, you have bought yourself a well-formed envelope — nothing more. The contents can still be empty, hallucinated, or quietly wrong.
Practitioners conflate three distinct failure layers: syntax, semantics, and orchestration. Each breaks differently, each requires different instrumentation, and each is routinely mistaken for the others. A 400 from your parser is not the same problem as a plausible-but-wrong customer_id, and neither is the same as a retry loop that resends poisoned context.
The symptoms cluster. Syntax failures are loud — the parser throws, the request fails, the dashboard spikes. Semantic failures are quiet — the pipeline completes, the record lands in the warehouse, and the corruption surfaces three weeks later when a downstream report shows a customer with an impossible lifetime value. Orchestration failures are the worst of both: they are loud in logs but ambiguous in cause, because a retry storm triggered by a semantic violation looks identical to one triggered by a provider timeout until you instrument the difference.
This article separates those layers, explains what constrained decoding actually does under the hood, and lays out a reliability stack that treats structured output as a versioned contract rather than a formatting hint. (See also: LLM output validation best practices.)
Why "Valid JSON" Is the Wrong Success Metric
Syntactic validity means the payload parses and conforms to the schema. Semantic correctness means the values are true, consistent, and useful. Task fidelity means the output actually advances the workflow it was generated for. These are three different bars, and most teams only measure the first.
The pattern is familiar. Your extraction agent returns {"items": []} — schema-valid, task-useless. Your classifier returns an enum member that exists in the schema but was hallucinated from a stale taxonomy. Your enrichment agent returns a phone number in the right field with the wrong country code. Every one of these passes validation.
Consider a concrete case. A schema declares "priority": {"enum": ["low", "medium", "high", "critical"]}. The decoder guarantees one of those four strings appears. It does not guarantee the model read the ticket correctly. A support-triage agent that labels a production outage "low" satisfies the contract at the syntax layer and violates it catastrophically at the semantic layer. Your validation suite is green. Your on-call rotation is not.
The empty-array failure deserves special attention because it is the most common and the most insidious. Models frequently learn that an empty result set is a safe answer when input is ambiguous, truncated, or outside the training distribution. The schema permits {"items": []}, so the decoder allows it, so the agent returns it, and the downstream consumer treats "no records found" as ground truth. Whether the array is empty because there are genuinely no records or because the model gave up is invisible to every layer that only checks shape.
Watch for the confidence trap, too. A model that emits confidence: 0.97 alongside a fabricated record is not telling you it is confident — it is emitting a token that fits the schema. Self-reported confidence is not a calibrated probability and should never be used as your fabrication detector. Replace it with external checks: cross-field invariants, retrieval grounding, and referential validation against your own systems of record.
A schema tells you the shape of an answer. It tells you nothing about whether the answer is true.
This is why the useful abstraction is the structured output contract: the schema plus semantic constraints plus validation plus a defined repair loop. The schema is one clause in a larger agreement. A contract specifies not just what fields exist but what values are admissible, what invariants must hold across fields, and what happens when the contract is violated. Treat the schema alone as the contract and you will keep shipping garbage that type-checks. (Related: semantic validation for LLM agents.)
How Constrained Decoding Actually Works
From Logits to Legal Tokens
At each generation step, a model produces a probability distribution over its vocabulary — the logits. Constrained decoding intercepts that distribution and masks tokens that would make the output illegal under the target grammar, renormalizing over the survivors. The sampler then chooses among legal continuations only. At each step, no sampled token can violate the compiled grammar, because illegal tokens have been removed from the distribution.
The enforcement is structural, not persuasive. You are not asking the model to please emit valid JSON; you are making invalid JSON unreachable relative to the grammar the runtime compiled.
That qualifier matters. Implementations differ. Some providers (notably OpenAI's Structured Outputs) compile your schema into a context-free grammar and mask at the token level. Open-source runtimes such as Outlines, guidance, and llguidance take related but not identical approaches, and each has its own supported-keyword set. The guarantee you get is always "conforms to the compiled grammar," which may be a lossy projection of your original schema. If the runtime silently drops a keyword, you have lost that constraint without any error.
The performance cost is real but usually acceptable. Masking adds per-token overhead — typically a few milliseconds per step depending on grammar complexity and vocabulary size — and grammars with large alternations or deep nesting can slow compilation enough to matter at high request volume. Cache compiled grammars per schema version; recompiling a complex schema on every request is a common and avoidable source of latency.
Schema-to-Grammar Compilation
To mask tokens, the runtime needs a formal grammar. Providers compile your JSON Schema into a state machine or context-free grammar that tracks where in the document the decoder currently is — inside an object, expecting a key, expecting a value of a specific type — and admits only tokens consistent with that state.
Some schema features compile cleanly and are widely supported:
- Types (
string,integer,boolean) map to token classes directly. - Enums become a fixed alternation of literal strings.
- Required properties and
additionalProperties: falsedefine exact key sequences. - Numeric bounds like
minimumandmaximumtranslate into range states.
Others are messier, partially supported, or rejected at compile time depending on the provider:
- Regex
patternsupport varies by runtime: some enforce a regex subset, some reject schemas containing it, and some silently ignore it. Always verify per provider with a concrete test — e.g., a phone-number pattern that should reject"abc". format(e.g.,email,date-time) is almost universally advisory. Do not assume it is enforced.- Nested conditionals (
if/then/else) andoneOfwith overlapping branches can blow up the grammar or degrade compile time. $refgraphs and recursive schemas vary widely in support.
The practical takeaway is a prioritization heuristic, not a law: features that compile cleanly are usually the ones that also constrain meaning best. Lean on enums, closed objects, and numeric bounds; treat regex and deeply nested unions as features to verify and test rather than assume. (See: choosing a structured output provider.)
A worked example of the lossy-projection risk: a schema declares "order_id": {"type": "string", "pattern": "^ORD-[0-9]{8}$"}. On a runtime that silently ignores pattern, the decoder will happily emit "order_id": "banana". The schema is satisfied at the type layer, the pattern is satisfied nowhere, and your downstream join against the orders table returns zero rows. The fix is not to trust the schema — it is a post-generation regex check plus a test in your CI suite that asserts the constraint actually fires on your chosen provider.
What the Decoder Cannot See
Here is the crux. The decoder enforces grammar, but the model still chooses which legal token comes next based on learned probability. Given a schema requiring "status": "active" | "inactive" | "pending", the model can only emit one of those three — but it may pick the wrong one, because nothing in the masking process knows which status is true.
Constrained decoding guarantees the model cannot write outside the lines. It does not guarantee the model writes the right thing inside them.
JSON Schema as a Contract, Not a Suggestion
Designing Schemas That Constrain Meaning
Every degree of freedom you leave in the schema is a degree of freedom the model can get wrong. Tighten aggressively:
- Enums over free strings for any categorical field — status, category, intent, currency.
- Explicit nullability rather than optional fields, so "unknown" is distinguishable from "absent."
additionalProperties: falseeverywhere, so the model cannot smuggle in invented fields.- Discriminated unions (
oneOfwith a requiredtypeconstant) rather than loose unions, so the branch is unambiguous and the grammar stays tractable.
The difference between "currency": {"type": "string"} and "currency": {"enum": ["USD", "EUR", "GBP"]} is the difference between a field the model can populate with "dollars" and a field it cannot. Multiply that by every field in a 40-property schema and you have the difference between an agent that produces usable records and one that produces plausible fiction. Tightening costs you a little design time and a lot of downstream pain.
Versioning the Contract
A contract you cannot version is not a contract. Pin a schema version in every request and persist it alongside the output. Adopt an additive-only compatibility policy: new optional fields and new enum members are backward-compatible; renames, removals, and narrowed enums are breaking and require a deprecation window. When an enum gains a member, your downstream consumers must already tolerate unknown values — otherwise a schema change becomes a production incident.
Persisting the schema version with the output is the part teams skip. Without it, a payload written in March is indistinguishable from one written in June, and when a bug surfaces you cannot tell whether the model was wrong or the schema was different. Store schema_version as a column, not a comment.
The Repair Loop
A defined repair loop is the difference between a contract and a wish. Specify it explicitly:
- Bound retries. One or two attempts, not an open loop.
- Feed back the validation error, not the full poisoned context. Resending the entire prior conversation amplifies hallucination; resending the specific violated constraint does not.
- Classify failures. Transient failures (truncation, timeout) are retryable. Schema mismatches and semantic violations are not — retrying them burns tokens and rarely converges. Route them to a dead-letter path with the request, the output, and the violated constraint attached.
- Trip a circuit breaker on repeated failure of the same input class so a bad schema change cannot melt your spend.
The failure-classification rule is the one most teams get wrong. A semantic violation retried three times does not converge on truth; it converges on whatever the model finds most probable after seeing the same ambiguous input again. You spend three times the tokens to arrive at the same wrong answer with more confidence. Dead-letter it, alert on the dead-letter rate, and fix the input or the schema.
Semantic Validation
Syntax is the floor. Add checks the decoder cannot perform:
- Cross-field invariants (e.g.,
end_date >= start_date). - Referential integrity (e.g.,
customer_idexists in your system of record). - Grounding (e.g., extracted facts appe