How to Tell If an AI Agent Finished the Task
The doubt you feel reading "done, all tests pass" is correct. Checking artefacts instead of assertions takes about three minutes.
"Done. I've implemented the feature and all tests pass."
You have read that sentence and felt a small doubt. The doubt is correct. An agent's claim that it finished is a claim about its own work produced by the thing that did the work, and it is the one output you should never take at face value. To tell whether an AI agent actually finished the task, you check artefacts rather than assertions: the diff, the test run, the places it did not touch, and the specific thing you asked for that reports in no summary.
The good news is that this takes about three minutes and almost all of it can be automated.
Why the summary is unreliable
Not because the model lies. Because of how the claim gets generated.
An agent that has run thirty tool calls summarises from a context window that may no longer hold the early ones. It reports a test suite passing based on the last run it remembers, which might predate its final three edits. It describes intent, and intent is what it has the best access to. A summary is a reconstruction, and the parts most likely to be missing are the parts that went wrong.
Three failure shapes recur.
Partial completion reported as whole. Four of five files changed, the fifth forgotten, and the summary describes the plan rather than the execution.
Tests passing because they stopped testing. An assertion loosened, a case skipped, a mock that now returns the expected value. Green, and meaningless.
The adjacent thing broken. The task is done and something it touched on the way is not. No summary mentions it, because the agent was not looking there.
How to tell if an AI agent finished the task, in three minutes
Do these in order. Each one is cheap and the order is deliberate: the fastest checks eliminate the most common failures.
Read the diff stat, not the diff.
git diff --statgenerates what git's documentation calls a diffstat: how many files changed and by how much. Compare it against your expectation before you formed one from the summary. A one-file change for a task you expected to span four is the strongest signal available, and it takes two seconds.Run the tests yourself, in a clean state. Not "did the agent say tests pass". Run them. The agent's claim and the result of
npm teston your machine are different kinds of fact.Diff the tests separately.
git diff --stat -- '*test*'is the single highest-value command in this list. If the task was a feature and the test files changed more than the source, something happened to the tests that you need to look at.Grep for the surrender markers.
skip,xit,todo,FIXME,eslint-disable,# type: ignore,as any, and a commented-out assertion. These are how green gets manufactured. In pytest the relevant decorators are documented under skipping:@pytest.mark.skiptakes an optional reason, and an xfail declares that you expect the test to fail, which is a quiet way for a real failure to stop counting.Check the thing you actually asked for. Name one observable behaviour before you start and verify that behaviour. Not the code, the behaviour.
Define finished before you start
Most of the ambiguity is created at the start, not at the end. An agent cannot overclaim against a definition of done it was given in checkable form.
Weak: "Add rate limiting to the API."
Checkable: "Add rate limiting to the API. Done means: a request from one IP over 100 in a minute gets a 429 with a Retry-After header; there is a test that asserts the 429 and a test that asserts a request under the limit still succeeds; no existing test is modified; curl run 101 times against the local server shows the 429 on the last call."
The second version produces a report you can grade, and it makes a dishonest summary difficult to write. This is the same discipline as reviewing an agent's plan before it runs, applied to the other end of the task. Writing it into a repository conventions file means you stop restating it every time.
Make the check mechanical
The checks above are the ones a human does. Most of them belong in a script, because a check you have to remember is a check you will skip at 6pm.
#!/usr/bin/env bash
# after-agent.sh: run before you believe the summary
set -u
echo "== files changed =="
git diff --stat
echo "== test files changed =="
git diff --stat -- '*test*' '*spec*'
echo "== surrender markers in the diff =="
git diff -U0 | grep -nE '^\+.*(\bskip\b|\bxit\b|FIXME|TODO|eslint-disable|type: ignore|as any)' || echo "none"
echo "== deleted assertions =="
git diff -U0 | grep -cE '^-.*(expect|assert)' | sed 's/^/removed assertion lines: /'
echo "== test suite, clean run =="
npm testThe deleted-assertions count is the one worth adding if you add only one thing. An agent that removed eleven assertion lines while adding a feature has told you something its summary did not.
A test suite the agent can run itself closes the loop earlier, which is why setting up a test suite an agent can run pays for itself quickly, whichever of the AI coding tools you have picked.
Signals that mean stop and read properly
Some outcomes justify more than three minutes.
Signal | Why it matters |
|---|---|
Test files changed more than source | The most common way green is faked |
Assertion lines removed | Deleted checks do not fail |
A new dependency you did not discuss | Scope expanded without a decision |
Unrelated files touched | Either a real coupling you did not know about, or drift |
Config or CI files changed | Can change what runs rather than what works |
A very short diff for a long task | Plan described, work not done |
Formatting churn across the repo | Hides the real change inside noise |
None of these prove failure. All of them mean read the diff rather than the summary.
What to do when it did not finish
Resist the urge to say "that's not finished, try again". It is the least informative instruction available, and it invites the agent to guess what you noticed.
Name the gap and the evidence: "The fifth file, billing/invoice.ts, was not changed, and test/invoice.test.ts has two assertions removed. Restore those assertions and make the change in that file without modifying any test." Specific feedback with the artefact named is how the second attempt beats the first. When a task reliably overruns, it is usually too big, and splitting it works better than more careful checking.
Frequently asked questions
Can I just ask the agent to verify its own work?
It helps and it does not replace the diff. A second pass with "list every file you changed and the test that covers each change" catches real omissions, because it forces enumeration rather than narration. It cannot catch what the agent never knew it broke.
What if the tests pass on my machine too?
Then check whether they still test anything. Run the test diff and the removed-assertions count. Tests that pass after being weakened pass on every machine.
Is a long session more likely to overclaim?
Yes. The longer the session, the more of the early work has fallen out of the usable context by the time the summary is written. Keeping context intact is covered in keeping an agent from losing context.
Should I commit before letting an agent work?
Always. A clean starting point is what makes every check in this post a one-line command, and it means abandoning a bad attempt costs nothing. Using git with an agent safely covers the rest.
How did this land?
About the author
Developer Advocate
Steve builds something with Swarmz every week and writes up what worked, what broke, and what he'd do differently. Tutorials and hands-on guides are his lane.
