TutorialsJSON SchemaLLM structured outputconstrained decodingOpenAI structured outputs

JSON Schema for Reliable LLM Structured Output: A Practical Tutorial

Stop JSONDecodeError in production. Learn JSON Schema patterns for reliable LLM structured output across OpenAI, Anthropic, and Gemini, with Python code.

Why Unstructured LLM Output Breaks Production Systems

Every team that ships an LLM feature hits the same wall. The demo works. Then production traffic arrives, and your json.loads() starts throwing JSONDecodeError at 2 a.m. The model wrapped its output in markdown fences. It added "Here is your JSON:" before the payload. It returned "quantity": "three" instead of 3, hallucinated a customer_id that doesn't exist, or got cut off mid-object by a token limit.

The failure taxonomy is predictable:

  • Markdown fencing — ```json wrappers that break naive parsers
  • Trailing or leading prose — "Sure! Here's the extraction you asked for:"
  • Hallucinated fields — keys you never defined, injected with plausible values
  • Wrong types — strings where integers belong, single objects where arrays belong
  • Truncated JSON — output hit max_tokens mid-structure
  • Silent coercion — your parser "fixes" a malformed value and corrupts downstream analytics

The downstream cost compounds fast: retry loops burn tokens and latency, pager alerts fire on ETL jobs, and corrupted records poison dashboards before anyone notices. The fix isn't better prompting alone. It's treating structured output as an interface contract between the model and your code — one that's machine-verifiable, not vibes-based.

This tutorial walks through a repeatable pattern for schema-valid structured output that works across providers, with runnable Python patterns and a production checklist.

What JSON Schema Actually Guarantees (and What It Doesn't)

JSON Schema is a declarative vocabulary for describing the shape of JSON documents. The keywords you'll use most:

  • type — object, array, string, number, integer, boolean, null
  • properties — the named fields and their subschemas
  • required — which fields must be present
  • enum — a closed set of allowed values
  • additionalProperties: false — reject any key you didn't define
  • Nested objects and arrays with items schemas

When an LLM API enforces a schema (more on enforcement methods below), you get syntactic validity, type safety, and enum constraints as hard guarantees. The output parses. The types match. Enum fields contain only allowed values.

What you don't get: correctness of the content. A schema can guarantee "sentiment" is one of ["positive", "negative", "neutral"]. It cannot guarantee the model read the text correctly.

Schema Validity vs. Semantic Correctness

A schema-valid record can still be wrong. If the source text says "I'm returning this product," the schema won't stop the model from emitting "sentiment": "positive" — the type is right, the enum is legal, the value is false. Schema enforcement eliminates parsing failures, not reasoning failures. Plan for both: schema validation in code, plus evaluation (golden sets, spot checks, or an LLM judge) for semantic accuracy. (See our guide on evaluating LLM outputs with golden datasets for building the accuracy side of that loop.)

One practical caveat: strict modes are stricter than the spec. OpenAI's strict Structured Outputs, for example, require additionalProperties: false on every object and require every property to be listed in required — optional fields are expressed as nullable types (["string", "null"]) rather than omitted from required. The supported keyword set is a subset of JSON Schema, and some keywords (certain format values, numeric bounds on some types) are unsupported or ignored. Anthropic's tool input_schema and Gemini's responseSchema have their own constraints and subsets. Always check the provider's current documentation before shipping a schema, and pin your SDK versions so a silent behavior change doesn't take you down.

How Major LLM APIs Enforce Structured Output

There are three distinct enforcement approaches, and they differ materially in reliability:

  1. Grammar-constrained decoding. The inference engine restricts sampling so that only tokens that keep the output valid under the schema (or a compiled grammar) can be produced. Implementations vary by provider, but the model is strongly steered away from — and in the strictest modes cannot produce — invalid JSON. This is the strongest practical guarantee.
  2. Function/tool calling with schema definitions. You declare a tool with a JSON Schema; the model returns a structured tool-call payload. Providers vary: some validate or bias generation toward the schema, some train the model to conform, and some apply constrained decoding under the hood. Client-side validation is still required.
  3. Prompt-only JSON mode. You ask in the prompt (and optionally set a JSON mime type). Output is usually valid JSON but not schema-conformant. Failure rates rise with schema complexity and output length — measure it on your own data before relying on it.

OpenAI, Anthropic, and Google: A Capability Comparison

ProviderFeatureEnforcement approachNotes
OpenAIStructured Outputs (response_format with json_schema, strict: true)Grammar-constrained samplingStrongest guarantee; strict-mode schema rules and a supported-keyword subset apply; check finish_reason/refusals
OpenAIJSON mode (response_format: json_object)Prompt-levelValid JSON, but no schema enforcement
AnthropicTool use (tools with input_schema)Trained conformance + prompting (no hard grammar guarantee)Reliable for extraction with client-side validation; input_schema is a JSON Schema subset
GoogleresponseSchema + response_mime_type: application/json in GeminiSchema-constrained generationresponseSchema is an OpenAPI 3.0 Schema subset, not full JSON Schema; supports propertyOrdering

Feature names, supported keywords, and constraints evolve quickly — link to live provider docs in your internal runbooks rather than copying this table. The buyer takeaway: enforcement method affects reliability, latency, and vendor lock-in. Grammar-constrained decoding is the most reliable but ties you to providers that offer it; tool-calling patterns are more portable but require client-side validation. (Compare this against our LLM provider selection checklist for the procurement angle.)

Tutorial: Building a Schema-Validated Extraction Pipeline

Our running example: extracting structured support-ticket records from messy inbound email text. Fields: ticket_id, category (enum), priority (enum), customer_email, summary, and line_items (array of objects).

pipeline diagram showing raw text → LLM with schema → validator → retry loop → structured store
pipeline diagram showing raw text → LLM with schema → validator → retry loop → structured store

Step 1 — Define the JSON Schema

Keep nesting shallow, close every enum, and forbid extra keys. This schema is written to satisfy OpenAI's strict-mode rules (every property in required, nullable for optional, additionalProperties: false everywhere) and is valid for jsonschema Draft 2020-12 validation.

TICKET_SCHEMA = {
    "type": "object",
    "additionalProperties": False,
    "required": [
        "ticket_id", "category", "priority",
        "customer_email", "summary", "line_items",
    ],
    "properties": {
        "ticket_id": {"type": "string"},
        "category": {
            "type": "string",
            "enum": ["billing", "technical", "account", "shipping", "other"],
        },
        "priority": {
            "type": "string",
            "enum": ["low", "medium", "high", "urgent"],
        },
        "customer_email": {"type": ["string", "null"]},
        "summary": {"type": "string", "maxLength": 500},
        "line_items": {
            "type": "array",
            "items": {
                "type": "object",
                "additionalProperties": False,
                "required": ["sku", "quantity"],
                "properties": {
                    "sku": {"type": "string"},
                    "quantity": {"type": "integer", "minimum": 1},
                },
            },
        },
    },
}

Design notes:

  • customer_email is ["string", "null"] rather than optional — strict modes require every key in required, so "optional" means "nullable."
  • summary has a maxLength to bound output tokens and reduce truncation risk.
  • quantity is integer with minimum: 1 — this is where "three" gets rejected, not silently coerced.
  • Enums are closed. If you need open-ended categories, add an explicit "other" and capture free text in a separate field.

Step 2 — Validate Before You Trust

Validate every response in your own code, regardless of provider guarantees. Provider-side enforcement can change, and tool-calling paths have no hard guarantee at all.

import jsonschema
from jsonschema import Draft202012Validator

VALIDATOR = Draft202012Validator(TICKET_SCHEMA)

def validate_ticket(obj: dict) -> list[str]:
    """Return a list of human-readable validation errors (empty = valid)."""
    errors = sorted(VALIDATOR.iter_errors(obj), key=lambda e: e.path)
    return [f"{'/'.join(map(str, e.path)) or '<root>'}: {e.message}" for e in errors]

Pin jsonschema (e.g., jsonschema>=4.18,<5) and pydantic>=2,<3 if you also model these records as Pydantic classes. Note the friction point: Pydantic's Optional[T] = None produces schemas where the field is not in required, which strict modes reject. If you generate schemas from Pydantic models, add a translation step that moves nullable fields into required with ["T", "null"] types.

Step 3 — Call the Model and Check for Truncation First

Before you parse, check the stop reason. A truncated response is not a parsing bug — it's a budget bug, and retrying with the same max_tokens will fail the same way.

import json
from openai import OpenAI

client = OpenAI()

def extract_ticket(raw_email: str, max_retries: int = 2) -> dict:
    messages = [
        {"role": "system", "content": "Extract a support ticket. Use only the provided schema."},
        {"role": "user", "content": raw_email},
    ]
    last_error = None
    for attempt in range(max_retries + 1):
        resp = client.chat.completions.create(
            model="gpt-4o-2024-08-06",  # pin a dated model snapshot
            messages=messages,
            max_tokens=1024,
            response_format={
                "type": "json_schema",
                "json_schema": {"name": "ticket", "strict": True, "schema": TICKET_SCHEMA},
            },
        )
        choice = resp.choices[0]

        # 1. Truncation check BEFORE parsing.
        if choice.finish_reason == "length":
            raise RuntimeError("Output truncated at max_tokens; raise the budget.")
        if choice.finish_reason == "content_filter" or choice.message.refusal:
            raise RuntimeError(f"Model refused: {choice.message.refusal}")

        # 2. Parse, then validate.
        try:
            obj = json.loads(choice.message.content)
        except json.JSONDecodeError as e:
            last_error = f"JSON decode error: {e}"
        else:
            errors = validate_ticket(obj)
            if not errors:
                return obj
            last_error = "Schema errors:\n" + "\n".join(errors)

        # 3. Feed the error back for a bounded repair attempt.
        messages.append({"role": "assistant", "content": choice.message.content or ""})
        messages.append({
            "role": "user",
            "content": f"Your previous output failed validation:\n{last_error}\n"
                       f"Return corrected JSON only, conforming to the schema.",
        })
    raise ValueError(f"Extraction failed after {max_retries + 1} attempts: {last_error}")

Why this shape matters:

  • Truncation is handled explicitly, not swallowed by a generic retry. finish_reason == "length" means the fix is a larger budget or a shorter schema/summary bound — not another identical call.
  • Refusals are surfaced, not parsed. OpenAI can return a refusal or an incomplete object; treat those as first-class outcomes.
  • Retries are bounded and error-informed. Sending the validator's error messages back is far more effective than blind retries, but cap it — an infinite repair loop is a cost incident.
  • Idempotency: if downstream writes have side effects, key them on a stable ticket_id (or a hash of the input) so a retry can't create duplicates.

Step 4 — Handle the Tool-Calling Path

For providers or models without strict structured outputs, use tool calling and validate client-side. The pattern is the same; only the envelope changes.

def extract_via_tool(raw_email: str) -> dict:
    resp = client.chat.completions.create(
        model="gpt-4o-2024-08-06",
        messages=[{"role": "user", "content": raw_email}],
        tools=[{
            "type": "function",
            "function": {
                "name": "emit_ticket",
                "description": "Emit a structured support ticket.",
                "parameters": TICKET_SCHEMA,
            },
        }],
        tool_choice={"type": "function", "function": {"name": "emit_ticket"}},
    )
    call = resp.choices[0].message.tool_calls[0]
    obj = json.loads(call.function.arguments)
    errors = validate_ticket(obj)
    if errors:
        raise ValueError("; ".join(errors))
    return obj

The tool-call arguments is a JSON string — parse it, then validate. Never assume the model honored input_schema just because it was declared.

Step 5 — Persist Only Validated Records

Write to your store only after validation passes. Keep the raw model output and the validation errors alongside rejected records for debugging and for building your evaluation set — those rejected samples are the most valuable data you'll collect.

Production Checklist

  • Pin model snapshots and SDK versions. gpt-4o-2024-08-06, not gpt-4o. Behavior changes are silent otherwise.
  • Validate in your code, always. Provider enforcement is a strong first line, not your only line — especially on tool-calling paths.
  • Check stop reasons before parsing. finish_reason/stop_reason, refusals, and content filters are distinct failure modes with distinct fixes.
  • Bound retries and back off. Error-informed repair, capped attempts, exponential backoff, and a circuit breaker.
  • Make writes idempotent. Stable keys so retries don't duplicate.
  • Bound output size. maxLength on free text, maxItems on arrays, and a max_tokens that comfortably exceeds your largest valid object.
  • Keep schemas shallow. Deep nesting raises latency and failure rates and hits provider subset limits.
  • Log raw output + validation errors for rejected records; feed them into a golden set.
  • Measure semantic accuracy separately. Schema validity is necessary, not sufficient — track extraction accuracy on a labeled set.
  • Centralize schemas as code. One source of truth, versioned, with a translation layer for provider-specific quirks (strict-mode required/nullable rules differ from Pydantic defaults).
  • Document the provider's supported keyword subset in your runbook and re-check it on every SDK upgrade.

Schema-validated structured output turns a flaky parse into a contract you can test, monitor, and reason about. Enforce where you can, validate everywhere, and treat semantic accuracy as a separate, measured concern.

ShareX / TwitterLinkedIn
← Back to Learn