How to Review a Multi-File Refactor From AI

Reading a 40 file multi-file refactor line by line is the wrong review, and most people do it anyway. Line by line is how you catch a typo. The failures that matter in a machine-produced refactor are not typos, they are structural: the one call site out of sixty that was missed, the behaviour that changed while the shape stayed the same, the test that was adjusted to accommodate a bug. None of those are visible at the line level, and all of them are visible in about fifteen minutes if you review in the right order.

The order is: shape, then invariants, then behaviour, then the parts you have not accounted for.

Why a large AI diff reads differently

A human refactoring sixty call sites gets bored and sloppy in a pattern you learn to predict. A model does not get bored, and its errors cluster differently.

It is unusually consistent. If the transformation is right, it is right everywhere, and spot-checking five sites tells you a lot about the other fifty-five. That is the good news and it is why line-by-line reading wastes most of its effort on confirmation.

It is also unusually willing to be locally reasonable. Faced with a call site that does not fit the pattern, a model will often make that one work in a slightly different way rather than stop and ask. The result compiles, passes, and quietly contains two conventions. Those sites are where the bugs live, and they are a small fraction of the diff.

And it will adjust tests. Not maliciously. A failing test looks like a thing to fix, and editing the test is a fix. This is the single most important thing to check and the easiest to find.

How to review a multi-file refactor from AI in four passes

Pass one: shape, two minutes

Before any code, look at the distribution of the change.

git diff --stat | tail -1
git diff --stat | sort -k3 -rn | head -20
git diff --stat -- '*test*' '*spec*'
git diff --name-only | sed 's|/[^/]*$||' | sort | uniq -c | sort -rn

You are asking three questions. Does the file count match the scope you expected? Are the changes evenly distributed, as a mechanical transformation should be, or is there one file with ten times the churn? Did tests change, and by how much relative to source?

An outlier file is the most reliable signal in a refactor. Even transformation plus one unusual file means that file did not fit the pattern. Read that one first, in full.

Pass two: invariants, three minutes

A refactor is defined by what must not change. State those and check them directly rather than looking for them while reading.

# Nothing was deleted that should not have been
git diff -U0 | grep -E '^-' | grep -vE '^---' | grep -cE '(expect|assert|if |throw|catch|await)'
# The old API is really gone, not half gone
git grep -n 'oldFunctionName' -- src/ test/
# Assertion count moved the way you expect
git diff -U0 | grep -cE '^\+.*(expect|assert)'
git diff -U0 | grep -cE '^-.*(expect|assert)'

Those commands lean on -U0, which git documents as generating diffs with zero lines of context, so each hunk contains only changed lines and a grep over it sees edits rather than their surroundings. That last pair is the check worth internalising. In a pure refactor, assertions added and removed should be roughly equal, and both close to zero. A refactor that removed nineteen assertions and added four has changed what your suite verifies, whatever its commit message says.

The git grep for the old symbol is how you find the missed call site. If the point was to remove oldFunctionName, it should appear zero times. One remaining occurrence, in a file nobody thought about, is the classic outcome.

Pass three: behaviour, five minutes

Now read code, and read only the places where behaviour could have moved.

• Error handling. Caught exceptions turning into returned nulls, or the reverse. Swallowed errors. A catch block that lost its rethrow.

• Boundary and default values. A default parameter that changed, an empty-collection case that now behaves differently.

• Order of operations. Validation that used to run before a write and now runs after.

• Async boundaries. A missing await, a promise no longer returned, parallel calls that were sequential for a reason.

• Type widening. as any, # type: ignore, a nullable that became non-nullable by assertion rather than by proof.

These five categories cover most real regressions. You can find them by grep as fast as by reading.

Pass four: the unaccounted, five minutes

Finish with what the diff does not explain.

Take the list of changed files and ask, for each one, which part of the task required it. Most will be obvious. The interesting ones are the files you cannot place. Sometimes they reveal a coupling you did not know existed, which is useful. Sometimes they are drift, and drift in a large diff is how unrelated changes ship unreviewed.

Then invert it: which files did you expect to change that did not? That list is where the missed call sites are, and it is the question no diff view asks for you.

The things to reject outright

Short list, no judgement calls needed.

Formatting churn deserves its reputation. A diff where every file has whitespace changes alongside logic is a diff nobody reviews properly, including you in six months when you bisect it. Ask for the formatting as a separate commit and review the logic against a clean base.

Make the next one easier

The best lever is upstream of the review. A refactor agreed in advance as a shape, and executed as a sequence of checkpointed commits, reviews in a fraction of the time of one 40 file drop. Reviewing the plan before it runs is where the shape gets fixed, and splitting a big task is what keeps each diff reviewable.

It is reasonable to use a second agent for a first pass, and worth knowing its limits: a model reviewing a diff is good at local consistency and poor at the unaccounted-files question, because it cannot know what you expected. Using an agent to review a pull request covers where that helps. Your verification habits around any agent's claim of completion are in how to tell if an agent finished the task, and the git workflow that makes all of this cheap is in using git with an AI coding agent safely. Diff review quality also varies by tool, which is part of picking an AI coding tool.

Frequently asked questions

Do I really not need to read every line?

Not in a mechanical refactor. Read every line of the outlier files, spot-check five instances of the repeated transformation, and spend your remaining attention on invariants and unaccounted files. That allocation finds more real problems per minute than sequential reading.

How many call sites should I spot-check?

Five, chosen for variety rather than at random: the simplest, the most complex, one in a module you know well, one in test code, and one from the outlier file. Consistency is the model's strength, so five similar results are genuinely informative about the rest.

What if the tests all pass?

Check what the tests now assert before you treat that as evidence. Added and removed assertion counts take ten seconds and change the meaning of a green run.

Should I ask the agent to explain its own diff?

Yes, as the last step rather than the first. Ask it to list every file and the reason that file changed. It is good at that enumeration, and comparing its list against your unaccounted-files list is a fast way to surface drift.