AI Coding Agents in a Production Incident: The Rules

Yes, use an AI coding agent during a production incident, and give it the diagnosis half only. It should read logs, bisect commits, correlate traces and write the timeline. It should not deploy, touch data, rotate credentials or change infrastructure while the site is down.

The split is not about how good the agent is. It is about reversibility. During an incident your error budget for a bad change is zero, because the normal safety net, noticing something is broken, is already spent. A wrong change at 3am does not look wrong. It looks like the incident continuing.

An incident has two halves and they have opposite requirements

Diagnosis is search. Read a lot, correlate, form hypotheses, discard most of them. It rewards breadth, tolerates being wrong repeatedly, and costs nothing when it fails because reading changes nothing.

Mitigation is a decision under uncertainty with real consequences. It rewards caution and a clear model of blast radius, and it punishes plausible-sounding action.

An AI coding agent is genuinely excellent at the first and structurally unsuited to the second, and the reason is worth stating plainly: an agent has no stake in the outcome and no sense of what is irreversible. It will propose truncating a table with the same steady tone it uses to propose adding a log line. Nothing in its behaviour distinguishes those, and during an incident your own ability to make that distinction is at its weakest too.

What to hand it, specifically

These are the jobs where an agent outperforms a tired human by a wide margin.

• Log triage across services. Ten thousand lines across four services, find every distinct error signature since 02:10 and count them. This is the single highest-value use, and it is pure reading.

• Diff bisection. Here are the 14 commits deployed in the last six hours and the symptom. Which of these could plausibly cause it, and why? Narrowing 14 to 2 in ninety seconds is the difference between a fast recovery and a slow one. This is what finding which change broke your build looks like under time pressure.

• Config and dependency archaeology. What changed in this service's environment variables between Tuesday and now? Agents are good at the boring cross-referencing that humans skip at 3am.

• Drafting the mitigation, not applying it. Write the patch, write the rollback for the patch, explain what it assumes. A human reads both and decides.

• Keeping the timeline. Have it log every hypothesis, check and action with timestamps as you go. You will not reconstruct this accurately afterwards, and it makes the postmortem an editing job rather than an archaeology project.

The reversibility test

One question decides everything else. If this action is wrong, can I undo it in under a minute without needing anything I do not already have?

If yes, an agent can do it behind a human approval. Deploying a one-line revert to a stateless service, toggling a feature flag, scaling a replica count.

If no, a human does it. Anything that writes to a database, deletes anything, rotates a secret, modifies DNS, changes IAM, or runs a migration. Data changes fail this test almost by definition: a bad write is not undone by reverting the code that made it, and an agent that deleted production data is not a hypothetical failure mode.

The test is useful precisely because it is answerable while exhausted. You do not have to evaluate the agent's reasoning, which is the hard thing at 3am. You only have to classify the action, which is easy.

A setup that makes this the default

Deciding all this mid-incident does not work. Wire it in beforehand. The principle is older than AI agents: Google's SRE book chapter on managing incidents puts it as "the operations team should be the only group modifying the system during an incident", and an agent is exactly the sort of uncoordinated second actor that rule exists to exclude.

1. Give the agent read-only production access, permanently. Logs, metrics, traces, a read replica. Set up in advance, not requested during an outage, because the worst time to be provisioning credentials is while customers are complaining. Our guide on read-only production access for coding agents covers the scoping.

2. Separate the write path entirely. Deploy credentials do not live in the agent's environment. Not scoped tightly, not guarded by a confirmation prompt. Absent. A confirmation prompt is a speed bump for a human and nothing at all for an agent that has been told to fix the problem.

3. Agree the incident roles before you need them. One human owns the decision to change production. The agent advises. If you are solo, you still hold the role; the point is that it is a role rather than a mood.

4. Keep a rollback that does not depend on the agent. A documented one-command revert you have actually run. If your recovery plan requires an agent that is currently confidently wrong, you do not have a recovery plan. More on this in rolling back a bad agent change.

The failure mode nobody warns you about

It is not the agent doing something catastrophic. It is the agent being persuasive.

An agent asked to diagnose an incident will produce a confident, well-structured explanation whether or not it has evidence for one, because that is what it was trained to produce. At 3am, with a page still firing, a coherent story is enormously attractive. You will want to believe it. And a wrong hypothesis presented well does not just waste time, it actively steers you away from the real cause, because you stop looking.

The countermeasure is cheap and you should build it into how you ask. Require evidence with every hypothesis: which log line, which metric, which commit. Ask explicitly for the two or three most likely causes rather than the cause, because a ranked list is honest about uncertainty in a way a single answer is not. And when the agent cannot support a claim, make it say so; forcing an I do not know is more valuable during an incident than at any other time.

The 3am solo founder counter-argument

The obvious objection: this is fine advice for a team with an on-call rota, but if you are one person and the site is down, the agent is all the help there is.

Fair, and the answer is not to loosen the rule but to lower the bar for what counts as mitigation. When alone, prefer the boring recoveries an agent can help you execute safely: roll back the deploy, turn the feature flag off, scale up, fail over to the last good image. These are reversible and they resolve a large share of real incidents. What you should not do alone at 3am is let an agent write a clever forward fix to a data problem, which is exactly the situation where a second pair of eyes was the only thing that would have caught it.

If the only available fix is irreversible, the right move is usually to wait until morning behind a maintenance page rather than to act tired with an agent that cannot tell you it is guessing. Downtime is recoverable. A corrupted table is a different category of Tuesday. Our notes on what to do when an AI-built app breaks in production cover the triage order for that decision.

Afterwards is where it earns its keep

The postmortem is the least controversial and most valuable use. An agent that watched the whole incident has the timeline, the commands run, the hypotheses discarded and the logs. It can produce a first draft in minutes that would take you an hour, and crucially it does it without the defensiveness a human writing about their own outage brings.

Two cautions. Have it separate what happened from why, because it will blend them, and check the timeline against the raw logs rather than against its own earlier summary. A postmortem built on the agent's recollection of its own reasoning is a story about a story.

Frequently asked questions

Can an agent be on-call?

It can watch, correlate and draft. It should not be the thing that decides an incident is resolved, and it should not be the only party notified. Alert routing to a human stays a human system.

What about agents that automatically fix failing CI?

Different risk class, and generally fine. CI is a sandbox where being wrong costs a red build. The rules here are about production, where being wrong costs customers.

Should I give the agent access to customer data during an incident?

Only in the form you would give a contractor: a read replica with personal fields masked, and only if the incident genuinely requires record-level inspection. Most do not; most need aggregate counts. The blast radius question applies to reads as well as writes.

Does this change if the codebase was written by AI in the first place?

It raises the stakes rather than lowering them. A codebase nobody has read end to end is one where blast radius is harder to predict, which makes the reversibility test more important, not less. For the wider set of practices, see our overview of AI coding tools.