How to Prompt AI to Fix a Flaky Test

You push again. The test fails. You push again with no code changes, and it passes. Every developer who has run tests in CI recognizes this exact moment: a red X that turns green on nothing more than a rerun. That is a flaky test, and it behaves nothing like a failing test. A failing test breaks the same way every single time you run it. A flaky test passes sometimes and fails other times against the exact same code. That distinction is the entire point of this post, because the obvious move, telling an AI coding agent to fix a test that behaves this way, can quietly make things worse instead of better.

If your test fails consistently, in the same way, on every single run, you're dealing with a genuinely broken test or a genuine bug in the code under test, and the fix is comparatively simple: find the failure and correct the code or the assertion. That case is covered in how to prompt AI to fix a failing test, which is the companion piece to this one. A flaky test is a different animal entirely. There is no single reliable reproduction, which means an AI agent cannot just read a stack trace and patch the obvious line. It faces pressure instead to make the red go green by whatever means are available, and that pressure is exactly where the danger lives.

An AI coding agent told to simply make a flaky test pass will often do precisely that and nothing more. It will wrap the test body in a retry loop. It will drop a sleep call in to buy more time before an assertion runs. It will widen a numeric tolerance or swap an exact match for a fuzzy one. The test goes green, the pull request merges, and the race condition, the shared mutable state, the bad timing assumption, or the flaky external call that caused the original failure is still sitting in your codebase, unchanged. You haven't fixed the bug. You've disabled the alarm that was telling you about it.

Diagnose before you prompt

Before writing a single prompt, spend a few minutes figuring out what kind of flakiness you're actually looking at. Most flaky tests trace back to one of a small number of well-known causes:

• Shared or global state that leaks between test cases, like a static variable, a singleton, or a database row a previous test didn't clean up

• Race conditions in async code, where two operations can resolve in either order depending on system load

• Reliance on wall-clock time, such as an assertion that assumes an operation finishes within an arbitrary number of milliseconds

• Test order dependency, where a test only fails when it happens to run after, or before, a specific other test

• Real network calls to external services that are themselves occasionally slow or briefly unavailable

• Unseeded randomness, where a random input occasionally hits an edge case the original test author didn't account for

This is not a rare annoyance. Google's engineering team has written about the scale of the flaky test problem, describing how enough unreliable failures teach engineers to distrust every red build, real failures included. That erosion is exactly why flaky tests show up so often as a complaint once a team wires up CI/CD for an AI-built app: a test that fails once in twenty runs looks harmless in isolation but can block a meaningful fraction of deploys once it runs on every commit.

What NOT to ask an AI agent to do

Once you've confirmed a test is flaky rather than failing, be precise about what you ask an AI coding agent to do. A few common phrasings reliably produce the wrong outcome:

• Asking it to "just make this test pass" gives the agent full license to hide the bug however is fastest, and fastest is rarely the correct fix.

• Asking it to add a retry or a retry loop as the first move can mask a race condition indefinitely. Unless the retry is logged and bounded, it just redistributes the failure rate instead of removing the cause.

• Asking it to add a sleep call or extend a timeout treats a symptom, something wasn't ready in time, as if it were the actual cause, and it will simply fail again on a slower runner or a busier machine.

• Asking it to loosen an assertion, a wider numeric tolerance, a partial match instead of an exact one, quietly narrows what the test actually verifies, so it stops catching the exact class of bug that made it flaky in the first place.

The same discipline applies across AI coding tools generally: a precise, scoped instruction tends to get a precise, scoped result, while a vague one gets whatever path requires the least effort from the model.

Prompts that actually work

The prompts that get useful results treat the agent as a debugging partner rather than a green-checkmark generator, and they ask for root-cause evidence before any code changes are proposed.

Run this test 50 times in a tight loop with verbose logging enabled, capture the full output from both a passing run and a failing run, and diff them to show exactly what differs, including timing, the order of log lines, and any external calls made.

Search this test file and the code it exercises for shared mutable state, static variables, singletons, or database records that aren't reset between runs, and tell me which ones could explain a test that passes when run alone but fails when the full suite runs together.

Trace every async operation in this test and in the code under test, find any place where an assertion runs before all promises or callbacks are guaranteed to have resolved, and show me the exact race condition without changing any test behavior yet.

Framing the request like an actual bug report, reproduction steps, expected versus actual behavior, whatever logs you managed to capture from a failing run, gives the agent something concrete to reason about instead of a vague complaint. How to write a bug report for an AI coding agent covers the format that tends to produce a real diagnosis instead of a guess.

When a retry really is the right fix

None of this means every retry is wrong. If a test genuinely depends on an external service, a third-party API, a flaky staging environment, a payment sandbox, and that service is occasionally slow rather than broken, a bounded retry with logging is a legitimate, documented engineering decision. The difference is intent and visibility. Three retries with a fixed backoff, a log line explaining why the retry exists, and a comment naming the specific dependency it compensates for, checked in deliberately, is not the same thing as an agent quietly wrapping a failing assertion in a while loop because you asked it to make the red go away. One is a tradeoff a team can see and remove later once the dependency improves. The other is a bug wearing a green checkmark.

FAQ

How do I know if a test is flaky or actually broken?

Run it against the exact same commit, with no changes at all, at least 10 to 20 times in a row. If it fails every single time, it's failing, not flaky, and the diagnostic approach in this post doesn't apply, use the failing-test playbook instead. If it fails some fraction of the time, even once in 30 runs, it's flaky, and the underlying cause is almost certainly one of the sources listed above rather than a simple logic error.

Is it bad practice to just retry a flaky test?

Retrying without investigating first is a bad habit because it hides the cause and lets the same bug resurface somewhere else later. A documented, bounded retry for a genuinely unreliable external dependency is standard practice in integration testing. The problem isn't retries themselves, it's treating retry as the default answer for every flaky test regardless of the cause.

Can AI coding agents cause flaky tests?

Yes. An agent generating tests quickly can introduce shared fixtures, unseeded random data, or timing assumptions that only show up as flakiness once the suite runs many times or in parallel. Reviewing AI-generated tests for exactly these patterns before merging catches most of it before it ever reaches CI.

How many times should I rerun a test to confirm it's flaky?

For a fast unit test, 20 to 50 runs in a tight loop is usually enough to get a reliable failure rate. For a slower integration test, 10 runs spread across a few separate CI jobs is more realistic. Reproducing even a 5 percent failure rate consistently is enough to call a test flaky rather than a one-off fluke.