Stop AI Agents Breaking Your API Contract

The agent renamed one JSON field from user_id to userId while tidying a handler. Every test passed, because the tests were generated from the same code. The mobile app, which nobody regenerated anything for, started showing blank profiles about four hours later.

The fix is not a better prompt. It is having a written contract that the agent is not allowed to edit, and a test that fails when the code drifts from it. Half a day of setup, and it holds for every future change whether a human or a model makes it.

What counts as breaking the contract

A change is breaking if an existing client that was working stops working, without that client changing. That is narrower than it sounds and it catches things people forget:

• Renaming or removing a response field, including changing its case.

• Making a previously optional request field required.

• Narrowing a type, for example accepting a string before and only an integer now.

• Changing the shape of an error, including its status code.

• Changing a default, so an omitted parameter now behaves differently.

Adding an optional field is not breaking. Adding a new endpoint is not breaking. Most safe changes are additive, which is a useful rule to give an agent directly.

The five breaks agents make most

Write the contract down somewhere the agent will read

A schema in a file beats a schema in your head, and it beats one inferred from code, because code changes are exactly what you are trying to detect. An OpenAPI document is the usual choice.

# openapi.yaml (excerpt)
components:
  schemas:
    User:
      type: object
      required: [user_id, email]
      properties:
        user_id:   { type: string, format: uuid }
        email:     { type: string, format: email }
        nickname:  { type: string, nullable: true }
      additionalProperties: false

Then say so in the file your agent loads on every task, in the imperative, with the path spelled out:

## API contract
`openapi.yaml` is the source of truth for every HTTP response shape.
Do not edit it to make code or tests pass.
Additive changes only: new optional fields and new endpoints are fine.
A rename, a removal, a narrowed type or a new required field needs a version bump,
and you should stop and ask before making one.

The test that actually fails

Instructions are a suggestion. A failing test is not. Validate a real response against the schema, so the check breaks when the code drifts rather than when someone remembers to look:

import yaml, jsonschema
from app.testing import client

SPEC = yaml.safe_load(open("openapi.yaml"))
USER = SPEC["components"]["schemas"]["User"]

def test_user_response_matches_contract():
    body = client.get("/v1/users/me").json()
    jsonschema.validate(body, USER)   # additionalProperties: false catches renames

def test_minimal_payload_still_accepted():
    # a client written before the newest field existed
    r = client.post("/v1/users", json={"email": "a@example.com"})
    assert r.status_code == 201

def test_error_shape_is_stable():
    r = client.get("/v1/users/does-not-exist")
    assert r.status_code == 404
    assert set(r.json()) == {"error", "message"}

Three tests, and they cover four of the five rows in the table. The point of additionalProperties: false is that a rename shows up as both a missing field and an unexpected one, so it cannot slip through as a tolerated extra.

Generating these with the model is fine, as long as you generate them from the schema and not from the implementation. Using AI to write tests goes into where that distinction matters most.

Versioning what genuinely has to break

Sometimes the change is correct and breaking anyway. The contract is not there to prevent that, it is there to make it deliberate. Three options, in increasing cost:

Most solo projects never need the third. The first covers renames, which is the most common accidental break, and it turns a four hour outage into a line in a changelog. Semantic versioning is the usual vocabulary for signalling which of these you did, and it is worth using even if the only consumer is your own mobile app.

Whichever you pick, write the deletion date down when you deprecate. A deprecated field with no removal date is a permanent field.

Where the schema should live

Two workable arrangements, and one that looks tidy and is not.

• Schema first. The OpenAPI document is written by hand and the code is checked against it. Best when clients are external or numerous, because the contract exists independently of any implementation.

• Code first with a frozen snapshot. The schema is generated from code, then committed and diffed on every build. The generated file is the artefact under review, so a change shows up as a diff a human reads.

• Generated fresh on every run. This is the one that looks tidy and provides nothing. If the schema is regenerated from the code at test time, it agrees with the code by construction and can never disagree, so it detects nothing.

The third arrangement is common in AI-generated projects, because generating the schema at build time is the obvious thing to do and nothing about it looks wrong. The check for it: change a field name and see whether anything fails. If not, your contract test is decorative. JSON Schema validation against a committed file is what makes it real.

Make the schema hard to touch

Agents follow the path of least resistance, and editing a schema is easier than fixing code. Close that path:

• Put the schema behind a CODEOWNERS entry so a change requires a human review.

• Add a CI step that diffs the schema against the base branch and fails on any non-additive change, so a bypass is visible rather than silent.

• Keep the schema outside the directories the agent normally works in, which pairs with keeping an agent out of unrelated files.

A pre-merge checklist

1. Did the diff touch the schema? If yes, read that hunk first and separately from the code.

2. Do the contract tests still pass without being edited? A test that changed in the same commit as the code it guards proves nothing.

3. Search the diff for renamed keys. Case changes are the ones that read as harmless and are not.

4. If something is genuinely breaking, version it rather than arguing with it. A new path is cheaper than a silent outage.

Once this is in place, running the agent unattended gets a lot less frightening, which is the setup described in running an agent in CI.

The same discipline outside the API

Response shapes are the visible contract, and three others break just as loudly with less warning:

• Database column names consumed by a report, an export or somebody's spreadsheet formula.

• Event and webhook payloads, which are an API with clients you cannot see and cannot notify.

• Environment variable names, where a rename means a deployment that starts cleanly and behaves as though a feature was switched off.

Each deserves the same treatment: a written definition the agent is told not to edit, and one test that fails when reality diverges from it. If you are exposing any of this deliberately, adding a public API to an AI-built app covers the design side, and the general habit of stopping unrequested changes reduces how often you need any of it.

Frequently asked questions

Is this overkill for an internal API?

The full CI setup often is. The schema file and one validation test are not, and they take about twenty minutes. Internal consumers break just as loudly, they just complain in a different channel.

What if the agent edits the schema anyway?

It will, occasionally. That is why the schema diff runs in CI rather than living only in an instruction file. Treat instructions as a way to reduce the frequency and treat the CI check as the thing that actually holds.

Should I generate clients from the schema?

If you control the clients, yes: it turns a runtime break into a compile-time one. It does not help with clients you do not control, such as a shipped mobile build, which is exactly where these outages hurt most.

Does versioning solve this on its own?

No. Versioning gives you a legitimate way to make breaking changes. It does nothing about the accidental ones, which arrive inside a patch that was supposed to be a tidy-up. You still need the test.