Dashboard

Keep AI Output Format Consistent Across Runs

Prose descriptions of a format produce roughly that format. A schema, one filled example and a validator that rejects drift produce the same shape every time.

Steve Jefferson
Steve Jefferson
Developer Advocate
29 October 20261 min read

If you want to keep AI output format consistent across runs, stop rewriting the instruction. The prompt is not where this is won. You win it with an output contract that a machine can check, and a retry when the output fails the check.

Describing a format in prose produces output that is roughly that format, and roughly is the problem when you are processing four hundred rows.

Why the format drifts when the prompt did not change

  • Format is made of tokens too. The model samples its way through the shape of the answer exactly as it samples the content. Nothing privileges your heading structure over your wording.

  • Prose descriptions are ambiguous. "Include a short summary" is a request for a number nobody supplied. One run gives you eight words, the next gives you forty, and both obeyed you.

  • Input length leaks into output shape. A long document invites a long answer and extra sections you never asked for. The prompt is constant; the request is not.

  • Nothing checks. Drift has no consequence, so it continues. This is the actual root cause, and the other three are details.

How to keep AI output format consistent across runs

  1. Show one filled example instead of describing the format. A single worked example pins field order, casing, punctuation and length in a way three paragraphs of instruction cannot. If you only do one thing from this list, do this one.

  2. Make the format machine-checkable. Fields rather than prose, so that a validator can give a yes or no answer. The mechanics are in getting JSON output from AI.

  3. Put numbers on every limit. Not "brief", but "one sentence, twenty words or fewer". Not "a few tags", but "exactly three". A constraint you cannot check is a preference.

  4. Drop the temperature for extraction work. Creativity is not a virtue when you are filling fields. If your tool exposes it, set it to zero for anything structured.

The output contract

Write the shape down as a schema, which doubles as your validator and your documentation:

json
{
  "type": "object",
  "required": ["summary", "category", "tags", "confidence"],
  "additionalProperties": false,
  "properties": {
    "summary":    { "type": "string", "maxLength": 140 },
    "category":   { "enum": ["billing", "bug", "feature", "other"] },
    "tags":       { "type": "array", "items": { "type": "string" },
                    "minItems": 3, "maxItems": 3 },
    "confidence": { "type": "number", "minimum": 0, "maximum": 1 }
  }
}

Three details in there do most of the work. additionalProperties: false stops the model adding a helpful extra field. The enum stops it inventing a fifth category on a hard case, which is the single most common drift in classification work. And minItems with maxItems equal to three enforces the count that "a few tags" never will.

Validate and retry, do not eyeball

python
def extract(text, attempts=2):
    last_error = None
    for _ in range(attempts):
        raw = call_model(PROMPT, text, extra=last_error)
        try:
            data = json.loads(raw)
            validate(data, SCHEMA)       # raises on any contract breach
            return data
        except (json.JSONDecodeError, ValidationError) as e:
            last_error = f"Your previous reply was rejected: {e}. Return only valid JSON."
    raise OutputContractError(text[:80])

Two things matter here. The retry includes the validation error, so the second attempt is told what was wrong rather than asked again blindly. And the function raises after two attempts instead of returning something half-right, which keeps a bad record out of your database.

Then log the rejection rate, because that is your drift metric. A couple of percent is normal. Twenty percent means your contract is wrong, not that the model is bad: usually a field that genuinely does not apply to some inputs, and the fix is to allow null rather than to argue with it.

When the output is prose, not fields

You cannot schema-check a paragraph, so constrain the structure around it. Specify the exact headings and their order, the number of sections, and a word budget for each. Then check mechanically: do the headings match the expected set, is each section inside its range, is anything empty.

That gets you a reliable shape. For whether the writing is any good, a separate pass against written criteria works better than trying to do both at once, which is the approach in prompting AI to grade against a rubric. Filling a fixed document without damaging it is its own problem, covered in prompting AI to fill a template.

Treat the example as source code

Your filled example is the specification. Everything downstream depends on its field order and phrasing, so it belongs in version control next to the schema, and changing it is a change to your output that deserves a review.

Keep a handful of real inputs with their expected outputs beside it. When you change the prompt, the example or the model, run them and diff. That is also what tells you quickly whether a contract survives a move between models, which is the portability question in writing prompts that work across AI models. For the broader discipline these habits belong to, see our prompt engineering guide.

Frequently asked questions

Is a longer, stricter prompt the answer?

No, and past a point it makes things worse. Long format instructions compete with your actual task for the model's attention. One example plus a schema is shorter and holds better than a page of rules.

What if the model keeps adding a field I did not ask for?

Set additionalProperties to false and let the validator reject it. Telling the model not to add fields works most of the time, which is exactly the problem with relying on it.

Should I retry forever until it validates?

No. Cap it at two attempts and fail loudly. An input that fails twice usually cannot be answered in your schema, and an unbounded retry turns an awkward record into a runaway bill.

How did this land?

About the author

Steve Jefferson
Steve Jefferson

Developer Advocate

Steve builds something with Swarmz every week and writes up what worked, what broke, and what he'd do differently. Tutorials and hands-on guides are his lane.

Share

Get the next post in your inbox

One email a month. Product updates, engineering posts, and the best of Built with Swarmz.

I agree to receive emails about AI building tips and Swarmz product news. Unsubscribe any time.