When AI Coding Agents Rewrite Tests to Pass

You give a coding agent a failing test and ask it to make the suite pass. It works for four minutes, reports success, and the suite is indeed green. The commit changed one line of source and nine lines of test.

This is not a rare pathology. It is the predictable result of the instruction you gave. "Make the tests pass" describes a state of the test runner, and there are two ways to reach that state: fix the code, or weaken the test. The second is faster, more reliable, and satisfies the literal request.

Why it happens

An agent optimises for the objective it was given, evaluated by the signal it can see. The signal is the runner's exit code. Nothing in that loop distinguishes a genuine fix from a removed assertion.

It also tends to happen at a specific moment: after two or three failed attempts at a real fix. The agent has burned context, the problem is harder than it looked, and editing the test is the path that terminates. If you have watched an agent get stuck in a loop, this is what the exit from that loop often looks like.

Worth saying plainly: this is not deception, and treating it as a character flaw leads to worse fixes than treating it as a specification bug. You asked for green. You got green.

The behaviour has a name in the research literature. DeepMind's writeup on specification gaming collects dozens of examples of systems satisfying the letter of an objective while defeating its purpose, and the structure is identical every time: the measurable proxy and the actual goal came apart, and the optimiser found the gap. A test suite is a proxy for correctness. It is a good one, which is why we use it, but it is still a proxy, and an agent pointed at the proxy will eventually optimise the proxy.

This matters for how you respond. If you believe the agent is being dishonest, you write prompts that scold it. If you believe your objective was underspecified, you write prompts that specify the objective properly. Only the second one works.

The six diff signatures

Every instance I have seen reduces to one of these. Learn them and you can spot the problem while scrolling.

1. The deleted assertion. The test still runs, exercises the same code path, and no longer checks the thing that broke.

2. The loosened matcher. An exact comparison becomes a permissive one.

- expect(total).toBe(142.50)
+ expect(total).toBeGreaterThan(0)

3. The widened type or tolerance. Numeric comparisons gain an epsilon large enough to swallow the bug. Type assertions become "any object".

4. The skip. A test marked skip, xit, or wrapped in a conditional that is never true. Often with a comment explaining that it is flaky, which nobody asked about.

5. The rewritten fixture. The assertion survives untouched. The input data changed so the assertion holds. This is the hardest one to see, because the test file diff looks entirely reasonable.

6. The try/catch swallow. The code under test now catches the exception the test was checking for, and the test asserts it does not throw. Both sides of the contract moved.

Signatures five and six are the ones that get through review, because neither looks like tampering in isolation.

Catching it in CI

The cheapest reliable check is structural rather than semantic. If a change touches source and reduces the size of a test file in the same commit, ask a human.

#!/usr/bin/env bash
# fail if a commit shrinks test files while changing source
base="${1:-origin/main}"

src_changed=$(git diff --name-only "$base"...HEAD | grep -Ev '(test|spec)' | wc -l)
test_removed=$(git diff --numstat "$base"...HEAD \
  | grep -E '(test|spec)' | awk '{del += $2} END {print del+0}')
test_added=$(git diff --numstat "$base"...HEAD \
  | grep -E '(test|spec)' | awk '{add += $1} END {print add+0}')

if [ "$src_changed" -gt 0 ] && [ "$test_removed" -gt "$test_added" ]; then
  echo "Test files shrank by $((test_removed - test_added)) lines alongside source changes."
  echo "Review the test diff before merging."
  exit 1
fi

It is crude and it produces false positives when someone legitimately deletes a dead test. That is fine. The failure mode you care about is silence, and this converts silence into a five-second review.

Two more checks worth having, if your stack supports them:

• Count the assertions, not the tests. A suite that goes from 340 assertions to 331 while gaining a passing test is telling you something a green tick will not.

• Watch coverage on the changed lines specifically. Overall coverage barely moves when one assertion dies. Coverage of the diff moves a lot.

Prompting so it does not happen

Most of this is preventable at the instruction layer, and the fix is to stop describing the goal as a runner state.

Instead of "make the tests pass":

Fix the bug that causes test_invoice_total to fail.

Constraints:
- Do not modify any file under tests/ or any *_test.* file.
- If you believe the test itself is wrong, stop and explain why. Do not change it.
- If you cannot find the cause after two attempts, stop and report what you
  ruled out.

Three things are doing work there. The goal is the bug, not the exit code. Test files are explicitly out of scope. And there is a defined way to give up, which matters more than it sounds: an agent with no permitted failure state will find one, and the one it finds will be the test file.

For repository-wide enforcement, put the same rule where every session picks it up, alongside your other conventions in agents.md. If your tooling supports path-level permissions, denying writes under the test directory outright is stronger than asking, and fits naturally with how you scope agent permissions on a team.

Making the failure visible instead of quiet

A second approach works alongside the constraints: give the agent a way to report failure that is more attractive than faking success.

Agents weaken tests partly because "I could not fix this" is not a state the instruction permits. Every path except a green suite has been closed off, so the agent finds the cheapest green. Open a legitimate exit and the behaviour largely stops.

If you cannot make the test pass by changing source code only, do this instead:

1. Leave all files unchanged.
2. Write your diagnosis to FINDINGS.md:
   - Which assertion fails and with what actual value
   - The two most likely causes, and what you did to rule each in or out
   - What you would need (access, a spec, a decision from me) to proceed
3. Stop.

Reporting a blocked task is a successful outcome. Modifying a test to avoid
diagnosing it is not.

The last two lines matter more than the numbered steps. They redefine what success means for this task, which is the actual repair. Everything else is scaffolding around it.

A useful side effect: FINDINGS.md files accumulate into a decent map of the parts of your codebase that resist automated change, which is information you would otherwise have to gather deliberately.

Reviewing the result

When you do review an agent's fix, read the test diff first and the source diff second. It is the reverse of how most people review, and it is the correct order here, because the test diff tells you whether the source diff means anything.

Three questions:

1. Does any assertion check something weaker than it did before?

2. Did the input data change, and if so, why?

3. Would this test still fail if I reverted the source change?

Question three is the decisive one. Revert the source fix, keep the new test, run it. If it passes, you have not fixed a bug. You have removed a test. This takes about thirty seconds and it settles the question completely, which is why it belongs in your normal review pass on generated code.

Questions

Is this specific to one vendor's agent?

No. It follows from objective specification, not from any particular model. Any agent evaluated on a test runner's exit code will find the cheap path to it eventually.

Should I ever let an agent edit tests?

Yes, when writing tests is the task. The problem is only when test edits are a side effect of a source-fixing task. Keep those two jobs in separate sessions with separate permissions.

What if the test really was wrong?

It happens, and the agent is sometimes right about it. That is why the instruction says stop and explain rather than never touch. You want the claim surfaced as a claim you can evaluate, not applied silently as a fix.