Dashboard

Your AI Agent Ran Up a Huge Bill: First Moves

Revoke the credential first, investigate second. The four shapes a runaway loop takes, and the controls that make a repeat impossible rather than unlikely.

Carlo Zuercher
Carlo Zuercher
Staff Engineer, Platform
29 October 20261 min read

What to do when an AI agent runs up a huge bill, in order: kill the credential, confirm the spend has stopped, and only then open the logs. Most people do that backwards. They start reading logs to understand the problem while the loop that caused it is still running, and the number keeps climbing during the investigation.

This is an incident, so treat it like one. Stopping the bleeding and finding the cause are two different jobs, and they happen in that order.

What to do when an AI agent runs up a huge bill: the first thirty minutes

  1. Revoke or rotate the API key. Not the agent. Stopping the process you know about is not the same as stopping the spend. There may be a second deployment, a queued job retrying, a cron that fires again in four minutes, or a colleague's laptop. The key is the one thing you can disable instantly, and it covers all of them at once. Rotate it rather than deleting it if other services share it, which is itself a problem to fix later.

  2. Watch the usage figures for five minutes. Do not assume. Providers report usage with a lag, so the number will keep rising for a little while after you have actually stopped it. What you are looking for is the rate flattening, not the total falling. If requests are still arriving, something still holds a valid credential.

  3. Write down the window. First unusual request, last request, total spend. You will need this for the logs, and if you intend to ask the provider for anything, you will need it in writing.

  4. Now read the logs. The cause is not going anywhere.

Finding the loop

Runaway agent spend almost always takes one of four shapes:

  • Retry without backoff on an error that will never succeed. A malformed request returns a 400, the wrapper treats any failure as retryable, and it asks again several times a second. This is the most common and the most expensive.

  • The agent reading its own output as new input. A loop where each step appends the previous answer to the context and asks again, with no stopping condition that ever becomes true. Cost grows quadratically here, because the prompt is also getting longer every pass.

  • A tool error the agent interprets as "try again differently". The tool is broken, so every attempt fails, so the agent reasons about it and attempts again. It looks like diligence and bills like a denial of service against yourself.

  • A scheduled job overlapping itself. The run takes longer than the interval, so instances stack. Nothing is looping in your code; you just have forty of them.

In the logs, three things identify which one you have: the request rate per minute, whether request bodies are identical or growing, and the token count per call. Identical bodies at a high rate is the first shape. Steadily growing bodies is the second. If you cannot answer those questions from your logs, that gap is the real finding, and logging AI API calls in production is the fix.

Can you get the money back?

Usually you have no entitlement to it. Worth saying plainly, because the internet is full of people implying otherwise. You authorised the calls, and a bug in your code is not a billing error on the provider's side.

That said, providers do sometimes grant one-off credits for a genuine runaway, at their discretion. What helps:

  • Contacting them within days, not at the end of the billing cycle.

  • A clear root cause in one paragraph, with the time window and the request count.

  • Evidence you have already fixed it, including the control you added.

  • A first offence on an account with normal usage history.

What does not help: arguing you were unaware of the pricing, asking before you know the cause, or framing it as their fault. Ask once, politely, with the facts. Then budget as though the answer is no, because it usually is.

The controls that make it impossible rather than unlikely

A fix that relies on the bug not recurring is not a fix. Every one of these is a hard stop that does not need a human to be awake:

  1. A provider-side spend cap, where one is offered. The only control that works when your own code is the thing that is broken.

  2. A maximum iteration count on every agent loop. Not a timeout, a counter. An agent that has taken thirty steps on a task you expected to take five is wrong, and should stop rather than persevere.

  3. A per-key daily token budget you enforce yourself, which fails closed. Count tokens as you go and refuse the call when the day's budget is gone. This is the control that saves you on a provider without spend caps.

  4. A capped maximum output length on every call. Output is the expensive half, as the arithmetic in what makes an AI model expensive shows.

  5. Alerts on request rate, not only on spend. Spend alerts arrive hours late because billing aggregates. A request-per-minute alert fires in the first minute.

  6. Retry logic that distinguishes retryable from permanent. A 400 is never retried. A 429 or a 503 is retried with exponential backoff and a hard attempt limit. The same thinking as rate limiting an AI-built app, applied to your outbound calls.

  7. A separate key per environment. So a development loop cannot spend your production budget, and so revoking one does not take the product down.

If the agent in question is a coding agent rather than something in production, the narrower version of control three is covered in setting a budget cap on an AI coding agent.

Do the worst-case arithmetic once

The useful exercise, and it takes two minutes. Take your agent's maximum request rate, multiply by the tokens per call, multiply by your rate per million, multiply by twenty four hours. That number is what an unattended loop costs you overnight.

Most people have never calculated it, and are shocked by both the figure and how little stands between them and it. If the answer is more than you could absorb, you need control one or control three before you need anything else. Treating runaway spend as an availability and safety problem rather than a finance problem is the right frame, and sits alongside the other failure modes in the overview of AI risks.

Frequently asked questions

Should I delete the API key or rotate it?

Rotate it if anything legitimate depends on it, because deleting takes your product down alongside the runaway. Delete it if you know it was created for the agent alone. Either way the goal is that the old value stops working within a minute.

The agent crashed on its own. Am I safe?

Not necessarily. A crash stops that process, not a queue of pending jobs, a retry wrapper, or a second instance. Confirm with the usage figures rather than with the absence of a running process.

How quickly will I even notice next time?

Whatever your slowest signal is. If your only alert is a monthly invoice, your detection time is up to a month. A request-rate alert takes an afternoon to add and changes that to minutes, which is the single highest-value thing to take from an incident like this.

How did this land?

About the author

Carlo Zuercher
Carlo Zuercher

Staff Engineer, Platform

Carlo works on the platform that turns prompts into running apps. He writes the engineering deep dives and the changelog notes worth reading.

Share

Get the next post in your inbox

One email a month. Product updates, engineering posts, and the best of Built with Swarmz.

I agree to receive emails about AI building tips and Swarmz product news. Unsubscribe any time.