How to Get JSON Output From AI Reliably

Your prompt says "Respond with valid JSON only." It worked in testing for two days. Then a user submits a slightly unusual input and the model replies:

Sure! Here is the JSON you requested:

{"priority": "high", "category": "billing"}

Let me know if you need anything adjusted.

Your parser throws. The request fails. Nothing about the prompt changed.

Getting JSON out of a language model reliably is a solved problem, but the solution is not better prompt wording. It is understanding that a prompt asking for JSON is a request the model usually honors, while schema enforcement is a constraint it cannot violate. Most teams spend weeks in the first category before discovering the second.

Here are the three ways structured output actually breaks, and the fix for each.

Failure 1: The conversational wrapper

The model returns correct JSON surrounded by explanation, apology, or markdown fences.

This happens because the model was trained heavily to be a helpful conversational assistant, and that training pulls toward framing an answer rather than emitting a bare object. Under an unusual input, the assistant reflex wins over your instruction.

The weak fix, which is worth doing anyway: move the format rule into the system prompt rather than burying it in the user message, and state it as an absolute. "Output a single JSON object. No prose, no markdown fences, no explanation." The distinction between system and user prompts matters here because standing rules belong in the layer that persists across turns.

The pragmatic fix: stop assuming a clean response and extract. Find the first { and the last } and parse what lies between. Five lines of defensive code that removes an entire class of incident.

The real fix: enforcement, covered below, where the wrapper becomes impossible rather than unlikely.

Failure 2: Malformed JSON

Trailing commas, unescaped quotes inside string values, truncated output, single quotes instead of double.

Two different causes hide here, and they need different responses.

Truncation is not a formatting problem, it is a length problem. The model hit its output token limit mid-object. Raise max_tokens, or reduce what you are asking it to return per call. If your output is a long array of items, request them in batches. This is straightforward once you recognize it, and easy to misdiagnose as a prompting failure. Checking the finish reason on the response tells you immediately which one you have.

Genuine syntax errors come from the generation process itself. As covered in how AI models work, the model emits one token at a time, each chosen from a probability distribution, with no ability to revise what it already wrote. Nothing in that loop validates syntax. A stray token early in a string value produces an object that cannot parse, and the model will happily continue building on its own broken output.

Retrying with the parse error fed back in works surprisingly often, and is worth having as a fallback layer. It is also strictly worse than making the error impossible.

Failure 3: Valid JSON, wrong shape

This is the dangerous one, because nothing crashes.

The model returns parseable JSON that is missing a required field, uses a value outside your allowed set, nests an object one level deeper than expected, or invents a plausible field name. "priority": "urgent" when your system only handles low, medium, and high. Your code reads it, stores it, and something downstream misbehaves days later.

Syntax validation will not catch any of this. Only schema validation will.

The escalation ladder

Work down this list until the failure stops. Each rung is stronger and more expensive to set up than the one above it.

Most production systems want the bottom two together: constrained decoding for structural correctness, plus your own validation for business rules the schema cannot express.

How enforcement actually works

Constrained decoding is worth understanding because it changes what kind of guarantee you have.

At each generation step the model produces probabilities across its whole vocabulary. Constrained decoding masks out every token that would break the required structure before a token is sampled. OpenAI's structured outputs documentation describes it plainly: with strict: true and a supplied JSON schema, schema compliance is enforced at the generation level, and the model cannot emit a token that would violate the schema.

Not "is unlikely to." Cannot. The invalid tokens are not on the menu.

Using it means accepting real constraints, and these trip people up:

• Every field must be marked required. Optional fields are expressed by making the type a union with null, then treating null as absent in your code.

• **additionalProperties must be false** on every object. The model can only produce keys you declared.

• Several JSON Schema keywords are unsupported, including allOf, not, if, then, else, dependentRequired, and dependentSchemas. Conditional logic has to move into your application code.

• The root must be an object, not a top-level anyOf.

• There are size ceilings: up to 5,000 object properties, 10 levels of nesting, and 1,000 enum values across all properties.

• Refusals are separate. When the model declines on safety grounds, the response carries a refusal field instead of schema-conforming output, so handle that branch explicitly rather than letting it fall through your parser.

Anthropic's equivalent path is tool use: define a tool whose input schema is your target structure and read the arguments from the tool call. Same idea, different surface.

A schema that behaves well

Design choices in the schema itself change your error rate more than prompt wording does.

Use enums instead of free strings wherever the set is closed. "category" as an enum of six values cannot come back as a seventh, and it removes an entire category of downstream normalization.

Name fields the way the content reads. ticket_priority gets better results than p1, because the model has strong priors about what a descriptive name should contain and none about your abbreviation.

Keep nesting shallow. Deep structures raise both token cost and error rate, and flatter shapes are easier to validate and to change later.

Add a field for uncertainty rather than forcing a guess. A nullable confidence or an explicit "unknown" enum member gives the model somewhere honest to go. Without one, a model with no good answer will still produce its best-scoring token, which is how confident fabrications get written straight into your database.

Ask for the reasoning field first if you want reasoning at all. Because generation is strictly left to right, a reasoning field placed before answer actually informs the answer. Placed after, it is a post-hoc narration of a decision already made.

Validate anyway

Enforcement guarantees your output matches the schema. It does not guarantee the values are correct, permitted, or safe to act on.

A schema cannot express "this customer ID must exist in our database" or "a refund over 500 needs review." Keep your own validation layer, and treat model output as untrusted input, which is the same posture you would take toward any AI-generated code before shipping it.

Log the failures you do get, with the input that caused them. Structured output failures cluster around specific input patterns, and three examples usually reveal the pattern faster than any amount of prompt rewriting.

Frequently asked questions

Why does the model add markdown code fences around the JSON?

Because it was trained on a great deal of technical writing where code is fenced, so fencing is a high-probability continuation when the content looks like code. Strip them defensively, or use schema enforcement, which prevents the fence tokens from being generated at all.

Is JSON mode the same as structured outputs?

No. JSON mode only guarantees syntactically valid JSON. Structured outputs with a strict schema additionally guarantee the shape matches what you defined. Valid JSON in the wrong shape is the failure mode that causes silent bugs, so the difference matters.

Does asking for JSON cost more tokens?

Slightly. Braces, quotes, and field names are all tokens, and repeated field names across a long array add up. Shorter field names save real money at volume, though not enough to justify unclear ones. The tokens explainer covers how the counting works.

Should I retry when parsing fails?

As a fallback, yes, feeding the parse error back in. As a primary strategy, no. Retries add latency and cost on exactly the requests already going badly, and they do not fix the wrong-shape failures that never raise an error.

What if my model or provider has no schema enforcement?

Use few-shot examples showing the exact output shape, extract defensively rather than parsing the raw response, validate against a schema in your own code, and retry once with the error attached. That combination gets you most of the way there without provider support.