Build a Test Suite an AI Coding Agent Can Run
Set up a test suite an AI coding agent can run: one command, an honest exit code, failures that name a file, and why a hook beats an instruction.
To set up a test suite an AI coding agent can run, you need three properties and nothing else: one command starts it, the exit code is honest, and a failure names the file and line that broke. Most suites fail the second or third, which is why an agent reports success on a change that does not work. Fixing that is a half-day of plumbing, and it pays off on every task afterwards.
There is a fourth question underneath, and it is not about the suite at all: whether the agent runs the tests when you are not watching. Instructions and enforcement are different mechanisms, and the docs are explicit about which is which.
One command, no prerequisites
The command has to work from a clean checkout with no undocumented setup. If running tests requires a local database, a seeded environment file and a service on port 5432 that someone started by hand in March, the agent will do what a new hire does: skip it and hope.
# the goal
make test # or: npm test, pytest, cargo test
# what it must not require first
cp .env.example .env && docker compose up -d && ./scripts/seed.shWrap the prerequisites in the command rather than in the README. A make test that starts its own containers, waits for health, seeds and tears down is slower to write once and correct every time after.
Where the environment genuinely cannot be reproduced locally, make that the headline of the test command's output rather than a silent skip, so the agent sees a refusal rather than a pass.
An honest exit code
This is where suites quietly lie. A non-zero exit on failure is the entire contract between your tests and anything automated. Common ways it gets broken:
A wrapper script that ends with
echo "done", so the script's exit code is the echo'sA test runner invoked inside a pipeline, where the shell reports the last command's status
A watch mode that never exits
A runner configured to treat no-tests-found as success, which turns a broken filter into a green run
set -euo pipefail # fail on error, including in pipelines
pytest -q # exit code is the suite's
# not: pytest -q | tee test.log # without pipefail, tee's status winsVerify it directly rather than assuming. Break one assertion on purpose, run the command, then echo $? and confirm you get a non-zero number. Put it back. That ninety-second check catches a problem that otherwise hides for months, and it is the same check behind did your AI coding agent actually run the tests.
Failure output an agent can act on
An agent reads your test output as text and tries to locate the problem from it. Output that says AssertionError: False is not true gives it nothing to work with, so it guesses, and guessing is how one failing test becomes four changed files.
Good failure output contains the test name, the file and line, the expected value, the actual value, and nothing else of note. Three practical moves:
Assert on values, not booleans.
assert resp.status == 200prints both sides.assert check(resp)prints neither.Fail fast for agent runs. A first failure with a clean trace beats two hundred cascading failures, so give the agent a
-xstyle command and keep the full run for CI.Turn off decorative output. Progress bars, colour codes and spinners arrive as escape sequences and waste the agent's context.
# the command you hand the agent
pytest -x -q --no-header -p no:cacheprovider
# the command CI runs
pytest -q --junitxml=results.xmlSpeed matters for a different reason than usual. The agent will run the suite several times in one task, so a four-minute suite is sixteen minutes of a ten-minute job. Split a fast unit tier from a slow integration tier and point the agent at the fast one by default.
Telling the agent versus making it happen
Write the command down where the agent reads it. Claude Code's memory documentation says a project CLAUDE.md should carry "build and test commands, coding standards, architectural decisions, naming conventions, and common workflows", and recommends specific phrasing, giving "Run npm test before committing" as the example rather than a vague "Test your changes".
The same page is candid about the limit of that mechanism: Claude "treats them as context, not enforced configuration", and "To block an action regardless of what Claude decides, use a PreToolUse hook instead." It adds that an instruction that must run at a fixed point, "such as before every commit or after each file edit", belongs in a hook, because hooks "execute as shell commands at fixed lifecycle events and apply regardless of what Claude decides to do".
That distinction is the practical answer to "the agent said it ran the tests". A line in a markdown file is a request. A hook is a gate. Use the file for the command and the conventions, and a hook for the one rule you will not negotiate, which is usually that tests pass before a commit.
The same logic extends to the pipeline, where nothing the agent says matters and only the run counts, covered in running an AI coding agent in CI.
Setting up a test suite an AI coding agent can run: the checklist
Property | How to verify it in two minutes |
|---|---|
One command | Clone to a new directory, run it, see tests execute |
Honest exit code | Break an assertion, run, check the exit status is non-zero |
Useful failures | Read one failure and ask if it names a file and a line |
Fast enough | Time the default command, aim under about sixty seconds |
No interactive prompts | Run it with stdin closed and confirm it still completes |
Deterministic | Run it three times and compare, since a flaky suite teaches an agent to rerun until green |
That last row is worth more than it looks. A suite that fails intermittently trains anyone, human or agent, to treat red as noise.
What this changes
With these in place, the agent's claim of success becomes checkable in one step, and your review shifts from reading its summary to reading the diff and the run. The follow-on habits are in what to do when an agent says it is done and writing a bug report an agent can act on, since a reproducible failing test is the best bug report there is. For choosing the tool that sits on top of all this, see the AI coding tools guide.
How did this land?
About the author
Developer Advocate
Steve builds something with Swarmz every week and writes up what worked, what broke, and what he'd do differently. Tutorials and hands-on guides are his lane.
