How to Review What an AI Agent Did Overnight
An agent's own summary describes what it intended. What you need first is what it actually touched, sorted by how much damage each change could do.
Reviewing what an AI agent did overnight is not the same job as reviewing a colleague's pull request, and reading it chronologically is the wrong approach. The agent's own account of its work is a summary of what it intended. What you need first is what it actually touched, sorted by how much damage each change could do.
Start at the edges of the task, not the middle.
Read the surface before the content
Before opening a single file, get two lists: everything that changed, and everything that ran.
git diff --stat main...HEAD
git log --oneline main...HEADNow compare that against the task you set. The question is not "is this code good". It is "why is this file in the list". A task to fix a date-parsing bug that touched the authentication middleware, the CI configuration, or a lockfile has done something you did not ask for, and that is where your attention goes first, before the parts that look like the work.
Three categories deserve a look every single time, regardless of the task:
Dependency and lockfile changes. An added package is a supply-chain decision. Check what it is, who publishes it, and whether anything in the codebase already did the job.
Configuration, CI and infrastructure files. A changed workflow file can alter what runs with what permissions on every future commit.
Anything touching authentication, permissions, payments or data deletion. Blast radius is the sorting criterion, and these four are the top of it.
Only after those do you read the actual feature work.
The three artifacts to demand
An overnight run should leave behind three things, and if your setup does not produce them it is worth the hour to make it.
A diff. Not a description of a diff.
A test result from a clean environment. Tests that passed in the agent's own session prove less than you would like, because the session may have state the next run will not.
A log of commands it executed. Particularly anything that touched the network, installed something, or wrote outside the working directory.
The third is the one most people lack and the one that answers the uncomfortable questions. If you cannot reconstruct what ran, you are reviewing the output of a process you cannot describe, and the right response is to fix the logging before you run it unattended again rather than to read the diff harder.
Checking the work itself
Once the surface is clear, four checks catch most of what gets through.
Did it change the tests to fit the code? The most common failure mode in unattended runs. A test that was asserting something specific now asserts something weaker, or has been marked skipped, and the suite is green. Diff the test files separately and read every assertion that got looser.
Does the fix address the cause or the symptom? An agent asked to stop an error will reliably find a way to stop the error, and catching an exception and returning an empty result does stop it. Read the fix against your own understanding of why the bug happened.
Is there work that was silently abandoned? Check for a half-implemented path, a function added and never called, or a TODO comment that marks the point where it gave up. These are easy to miss because they are additions, and additions read as progress.
Does it run? Actually run it, against the real case that prompted the work. This sounds too obvious to list and it is the step most often skipped when the diff looks reasonable.
If the agent produced a plan before it started, read that plan against the diff as well. The gap between the two is usually where the interesting decisions were made, which is the main argument for reviewing an agent plan before it runs rather than only after.
Time-box it, and know when to throw it away
Set a limit before you start. Thirty minutes for a routine overnight run is reasonable. If you reach the limit and still do not understand what happened, that is a result, not a failure of concentration.
At that point you have two options, and only two. Discard the branch and re-run with a narrower task, or sit down and work through it properly as a piece of unfamiliar code you have inherited. What you must not do is merge something you have given up on understanding because it passes tests and the morning is getting on. The question of whether to fix or regenerate is usually settled by exactly this: how long it would take you to trust it either way.
A useful rule of thumb from running these: a diff you cannot summarise in three sentences after twenty minutes is a diff that was given too large a task. The problem is upstream, in the instruction, and re-running with a tighter scope is faster than finishing the review.
Unattended work concentrates a lot of exposure into one unobserved window, which is why it earns a place in the practical guide to AI risks rather than being purely a productivity question.
Set it up so the morning is cheap
The quality of the review is decided the night before.
Scope the task so the expected surface is small, and say explicitly which files or areas are in scope. Then an out-of-scope change is visible instantly rather than requiring judgement.
Run it against a branch, never the main line. Obvious, routinely skipped under time pressure.
Give it a [budget cap](https://swarmz.net/blog/how-to-set-a-budget-cap-on-an-ai-coding-agent), which also bounds how much work there is to review.
Restrict what it can reach. An agent that cannot touch production has a much shorter list of ways to surprise you, which is the whole argument for sandboxing. It should certainly not have write access to a production database.
Ask for a written note of anything it decided that you did not specify. Agents make dozens of small choices. A list of them is cheap to generate and turns an archaeological exercise into a conversation.
FAQ
Should I read every line of an overnight diff?
No, and trying is how people stop reviewing properly. Read the full diff on anything in the blast-radius categories, and skim the rest with a specific question: does this match what I asked for?
What if the tests pass and I still do not trust it?
Trust the instinct and find out why. Usually it is a change outside the expected surface, or a test that got weaker. If you cannot locate the reason within your time box, re-run with a narrower task.
How do I stop the agent changing tests to make them pass?
Say so in the instruction, keep test files out of scope where you can, and diff them separately so a change is never invisible. Treat any loosened assertion as a failed run.
Is overnight unattended work a good idea at all?
For well-scoped, verifiable tasks with a real test suite, yes. For open-ended work on code without tests, you are reviewing in the morning what you would have understood while writing, and usually losing the trade.
How did this land?
About the author
Developer Advocate
Steve builds something with Swarmz every week and writes up what worked, what broke, and what he'd do differently. Tutorials and hands-on guides are his lane.
