How to Use AI to Write Tests That Catch Bugs

Here is a function with a bug in it.

def apply_discount(price, code):
    if code == "SAVE10":
        return price * 0.9
    return price

The requirement, written down in a ticket somewhere, says discount codes are case-insensitive. This implementation is not. Now watch what happens when you select the function and ask an AI assistant to write tests for it:

def test_apply_discount_valid_code():
    assert apply_discount(100, "SAVE10") == 90

def test_apply_discount_invalid_code():
    assert apply_discount(100, "save10") == 100

def test_apply_discount_no_code():
    assert apply_discount(100, "") == 100

Three tests. All passing. Coverage report looks great. And the second one has just written the bug into your test suite as intended behavior, where it will now block anyone who tries to fix it.

This is the central failure mode of AI-generated tests, and it is not a model quality problem. It is a straightforward consequence of what you asked for.

Why this happens

When you point a model at a function and ask for tests, it derives the expected values from the code, because the code is the only thing you gave it. It has no access to the ticket, the spec, or the conversation where someone decided codes should be case-insensitive.

A test derived from an implementation can only ever assert that the implementation does what it does. That is a tautology dressed as verification. It cannot detect a gap between what the code does and what it is supposed to do, because it was never told what it is supposed to do.

Such tests are not worthless. They function as regression detection: if someone later changes the behavior accidentally, the tests fail. That is genuinely useful. It is just a much smaller claim than "these tests verify the code is correct," and the coverage percentage does not distinguish between the two.

Framing this in the standard terms, the tests are coupled to the implementation rather than the behavior. Martin Fowler's testing guide and practical test pyramid cover why implementation-coupled tests break under refactoring while proving little about correctness. AI assistants default to producing exactly this kind, at speed and in volume.

The fix: give it the requirement, not the code

The single change that matters most is inverting the input.

Instead of "write tests for this function," describe the intended behavior and let the model write tests against that description, ideally before it sees the implementation at all.

Write pytest tests for a function apply_discount(price, code).

Requirements:
- Code "SAVE10" gives 10% off
- Codes are case-insensitive
- Unknown or empty codes return the price unchanged
- Price must be non-negative; negative price raises ValueError
- Discounts never produce a negative result

Do not assume an implementation. Test only these requirements.

Now the case-insensitivity test asserts apply_discount(100, "save10") == 90, and it fails against the current code. That failure is the entire point. The test found the bug, which is the job.

This is closer to specification-driven testing than to code coverage, and it is where AI assistance is genuinely strong: you supply the requirements, which is the part requiring judgment, and it supplies the exhaustive enumeration of cases, which is the part humans get bored doing.

Where AI is actually better than you

Once the requirements are in the prompt, lean on the model for the thing it does well: thinking of inputs you would not.

Ask directly. "What edge cases does this specification not address?" A good model will come back with an empty string, a null, a price of exactly zero, a floating point value that rounds badly, a very large number, unicode in the code field, and whitespace padding. Some are irrelevant. Two or three usually are not, and those two or three are where production incidents come from.

This is a different task from writing tests, and worth running as its own step. Adversarial enumeration against a spec is close to the best use of a coding assistant. It has read an enormous amount of code and the failures that code produced, and it does not get tired at case eleven.

Boilerplate is the other clear win. Fixtures, parametrized case tables, mock setup, test data builders: mechanical, repetitive, and error-prone by hand. Hand all of it over, and see how to prompt AI to generate realistic test data for the fixtures and edge cases specifically.

Four rules that keep the tests honest

Make new tests fail first. A test written against a requirement should fail on code that does not meet it. If a freshly generated test suite passes completely on the first run, be suspicious rather than pleased. Either the code is genuinely correct, or the tests were written from the code. Break something on purpose and confirm the suite notices.

Never accept an assertion you have not read. This is the rule people break under time pressure, and it is the one that matters. An assertion is a claim about correct behavior. Skimming a generated test file and merging it means adopting claims you did not evaluate. Read each expected value and ask whether you agree with it. This is the same discipline that applies to reviewing any AI-generated code before shipping, and test files get less scrutiny precisely because they feel lower risk.

Treat coverage as a floor, not a score. AI makes 90% coverage cheap and easy, which makes coverage a much weaker signal than it used to be. A suite can cover every line while asserting almost nothing meaningful. Ask instead: if I introduced a plausible bug here, would anything go red?

Watch for over-mocking. Models mock generously, and a test where every dependency is mocked verifies that your function calls the mocks in the expected order. That is a test of the implementation's internal shape. It will break every refactor and catch few real defects.

Bugs AI tests reliably miss

Worth knowing where the blind spots are, because they line up with the same reasons AI-generated code fails on real projects.

Anything requiring knowledge outside the file: an invariant enforced elsewhere, a downstream service's expectations, a business rule that lives in someone's head.

Integration behavior. The model tests the unit it was shown. Most production failures happen between units, in the wiring, the serialization, the network boundary.

Concurrency. Race conditions do not appear in single-threaded test runs, and models rarely propose the interleaving that breaks things.

Performance and scale. A test passing on three records says nothing about thirty thousand.

Anything where the model shares your misconception. If you got the requirement wrong in the prompt, the tests will confirm your wrong requirement with total confidence.

A workflow that works

1. Write the requirements as plain bullet points. This is your job and takes five minutes.

2. Ask for edge cases against those requirements, before any tests. Add the good ones to the list.

3. Generate tests from the requirement list, explicitly instructing the model not to assume an implementation.

4. Run them against your current code. Read every failure and decide, one at a time, whether the test is wrong or the code is.

5. Fix whichever is actually wrong.

6. Have the model generate the boring parametrized variations once the core assertions are settled.

Step 4 is the part that cannot be delegated, and it is where the value is. Every failure is a discrepancy between what you said you wanted and what the code does, which is exactly the information you were trying to buy.

If your tests validate structured model output rather than ordinary functions, the same principle applies with an extra layer: assert against the schema and the business rules separately, as covered in getting reliable JSON output from AI.

Frequently asked questions

Should I let AI write all my tests?

Let it write most of the code in your tests. Do not let it decide what the tests should assert. The assertions encode your definition of correct, and that has to come from you.

Is AI-generated test coverage misleading?

It can be, badly. Coverage measures lines executed, not behavior verified. A suite generated from your implementation can reach very high coverage while being incapable of detecting that the implementation is wrong.

What is the fastest way to check whether my AI tests are any good?

Mutation testing, informally: change a comparison operator or a return value on purpose and rerun. If the suite still passes, it is not testing what you thought it was.

Do AI assistants write better tests for some languages?

Generally yes, in proportion to how much code in that language and framework was available during training. Popular stacks with strong testing conventions get noticeably better output than niche ones, which is part of the broader trade-off in picking AI coding tools.

Should tests be written before or after the implementation when using AI?

Before is meaningfully better here, more so than in manual development. Tests written before the implementation cannot be derived from it, which structurally prevents the failure this article is about.