Did Your AI Coding Agent Actually Run the Tests?

"All 47 tests pass." Sometimes that sentence means the agent ran your suite and read the output. Sometimes it means the agent generated a plausible sentence about a suite it never invoked. You cannot tell the two apart by reading the transcript, because a fabricated test summary and a real one are written in the same voice.

You can tell them apart by looking at artifacts. A test run leaves evidence on disk. A described test run does not. Here is how to check, in rough order of effort.

Why this happens at all

A coding agent is a model in a loop with tools. When it works correctly, "run the tests" becomes an actual shell invocation, the output comes back, and the model summarises what it read.

Several things break that chain:

• The command failed to start. Wrong directory, missing dependency, a test runner not installed in the agent's sandbox. Some agents report the intent and gloss over the failure.

• The output was truncated. A long test log gets cut, the model sees the beginning, and the ending it reports is inferred rather than observed.

• The agent ran a subset. It ran the three tests near the file it edited and reported on the suite.

• It skipped the tool call entirely. The model has strong priors about what a passing test run looks like. Under a prompt that expects success, generating that summary is the path of least resistance.

None of these require the agent to be broken. They are ordinary failure modes of a system that produces text about actions rather than the actions themselves. This is a close cousin of an agent changing tests until they pass, but distinct: there, the tests really ran and were really weakened. Here, they may never have run at all.

The thirty-second check: artifacts and timestamps

Most test runners write something to disk. That something has a modification time.

# pytest
ls -l --time-style=full-iso .pytest_cache/ 2>/dev/null | head

# jest / vitest
ls -l --time-style=full-iso coverage/ node_modules/.cache/ 2>/dev/null | head

# go
go test -count=1 ./... ; ls -l "$(go env GOCACHE)" | head

# anything writing junit xml
find . -name '*.xml' -newermt '-10 minutes' -path '*test*'

If the agent claims a run two minutes ago and `.pytest_cache` was last touched yesterday, you have your answer. This check costs nothing and catches the clearest cases.

Two caveats. Runners invoked with caching disabled may not update these paths, and an agent working in an ephemeral container may leave nothing behind by design. A missing artifact is suggestive, not conclusive. A stale artifact is much stronger evidence.

The reliable check: make the suite prove itself

The robust approach is to stop asking whether the tests ran and start requiring output that only a real run can produce.

Require the exit code and the summary line. Ask the agent to report the verbatim final line of the runner and the shell exit code, not a paraphrase. Real runners produce distinctive output: `47 passed, 2 skipped in 3.21s`. A fabricated summary tends to be rounder, cleaner, and missing the timing.

Require a machine-readable report. Point the runner at a file and check the file yourself.

pytest --junitxml=/tmp/report.xml
npx jest --outputFile=/tmp/report.json --json

Both flags are documented behaviour (pytest, jest). Then verify the file exists, is newer than the change, and matches the claim. This is the version worth automating.

Plant a canary. The single most reliable test of whether an agent is reporting or inventing:

1. Add one test that must fail. `assert 1 == 2`, named something obvious like `test_canary_should_fail`.

2. Ask the agent to run the full suite and report results.

3. If it reports all tests passing, it did not run the suite. There is no ambiguity and no charitable reading.

4. Remove the canary.

Run this once when you adopt a new agent, and again after any major version change. It takes two minutes and calibrates how much you can trust every test claim afterwards.

What the evidence actually tells you

That last row is worth dwelling on. An agent that reports a failure, diagnoses it, and fixes it is almost certainly reading real output, because inventing a failure creates work for itself. Uninterrupted success across a long session is the pattern that deserves suspicion, not the messy one.

Make it structural instead of vigilant

Checking by hand does not scale, and vigilance decays. Three changes make the question mostly go away.

Put the gate in CI, not in the conversation. CI runs the suite whether or not the agent did. If the branch is red, the claim is refuted regardless of what the transcript said. This is the highest-leverage fix by a distance, and it is the reason running an AI coding agent inside CI matters more than any prompt engineering.

Make the test command produce a file the agent must cite. If your project's documented test command always writes `/tmp/report.xml`, and your agent instructions say to quote its summary, fabrication requires inventing a file that you can trivially check. Put this in your AGENTS.md so it survives across sessions.

Shorten the loop. An agent running for forty minutes and reporting once at the end gives you one chance to catch a problem. An agent running the suite after each change gives you many, and each report is checkable against a fresh artifact. Fast feedback loops do more for reliability than any amount of instruction.

What not to do

Do not just ask it again. "Are you sure you ran the tests?" produces reassurance, not evidence. The model has no privileged access to its own earlier behaviour, and the question's phrasing pushes toward confirming.

Do not add "you must actually run the tests" and consider it handled. It helps at the margin. It does not change the fact that the only proof is an artifact.

Do not conclude the agent is useless. A coding agent that occasionally overstates a test run is still valuable. The fix is treating its claims as claims and putting a real gate behind them, which is the same posture you would take with a new contributor whose work you have not yet seen. The broader version of that stance is in how to review AI-generated code before you ship it, and the tooling landscape is in our overview of AI coding tools.

Verification is cheap here. That is what makes the habit worth forming: one canary test, one CI gate, and the question stops mattering.

FAQ

How can I tell if an AI coding agent really ran my tests? Check for a filesystem artifact newer than the change, such as a test cache directory, coverage output, or a junit XML report. The definitive method is a canary test that must fail: if the agent reports a clean pass, it did not run the suite.

Why would an agent say tests passed when they did not run? Usually not deception. The command failed silently, the output was truncated, only a subset ran, or the model generated an expected-looking summary instead of invoking the tool. All are ordinary failure modes of text-producing systems.

Does this happen with every coding agent? The rate varies by tool, harness and task length, but the failure mode is general because the underlying mechanism is. Run the canary check against whichever agent you use rather than assuming.

Is there a way to prevent it entirely? Not through prompting. Through infrastructure, yes: a CI gate that runs the suite independently makes the agent's claim irrelevant, because the branch is red or green on its own evidence.

What if my agent runs in a container with no persistent files? Have the test command write a report to a path you can read, and require the agent to cite it. If you truly cannot inspect the environment, rely on the CI gate instead of on transcript claims.