AI API Rate Limits Explained

Your app worked all week and started returning 429 errors this morning. Nothing changed on your side. The usual response is to add a retry, which sometimes fixes it and sometimes makes it dramatically worse.

The reason is that "rate limit" is four different limits wearing one error code, and the correct response to each is different. Telling them apart takes about a minute once you know what to look for.

The four limits

Requests per minute (RPM). How many calls you can make, regardless of size. A thousand tiny classification calls will hit this long before they hit anything else.

Tokens per minute (TPM). How much text you can push through, regardless of how many calls it takes. A handful of calls with large documents attached will hit this while your RPM sits near zero.

Concurrency. How many requests can be in flight at once. This one bites when you parallelise: twenty simultaneous calls may be refused even though twenty calls per minute is fine.

Daily or monthly caps. A budget ceiling rather than a flow control. Usually tied to your account tier or spend limit, and it does not reset in sixty seconds.

The trap is that RPM and TPM pull in opposite directions. Hitting an RPM limit suggests batching more work into fewer calls. Do that with large inputs and you walk straight into the TPM limit. Hitting TPM suggests splitting work across more calls, which walks back into RPM. Optimising for one without measuring the other is how teams spend a week going in circles.

Reading the error properly

Most providers tell you which limit you hit, in the response headers rather than the message body. Both OpenAI and Anthropic document their header sets, and the shape is broadly consistent across vendors. Look for headers along these lines:

x-ratelimit-limit-requests: 500
x-ratelimit-remaining-requests: 0
x-ratelimit-reset-requests: 12s
x-ratelimit-limit-tokens: 30000
x-ratelimit-remaining-tokens: 24180
x-ratelimit-reset-tokens: 0s

That example is unambiguous: requests exhausted, tokens barely touched. You have an RPM problem, and batching will help. If the numbers were reversed, batching would be exactly the wrong move.

Log these headers on every 429 rather than just the status code. It is a two-line change and it converts a recurring mystery into a fact.

Also check the retry-after header if there is one. A server telling you when to come back is more accurate than any backoff schedule you invent.

The fix for each

The last row deserves emphasis. Retrying against a daily cap does nothing except consume your own capacity and, on some providers, count against you. Detect it and fail cleanly instead.

Backoff that behaves

If you take one thing from this: retries need jitter. Without it, every request that failed together retries together, and your recovery attempt is a synchronised second wave.

import random, time

def call_with_backoff(fn, attempts=5, base=1.0, cap=30.0):
    for i in range(attempts):
        try:
            return fn()
        except RateLimited as e:
            if getattr(e, "retry_after", None):
                delay = e.retry_after
            else:
                delay = min(cap, base * (2 ** i))
                delay = delay * (0.5 + random.random() / 2)   # jitter
            if i == attempts - 1:
                raise
            time.sleep(delay)

Two details matter. The provider's retry-after wins over your own calculation whenever it is present. And the jitter multiplier spreads the retry across a window rather than firing everything at the same instant.

For anything beyond a handful of calls, put a queue in front instead of retrying inline. A token bucket sized just below your published limit turns rate limiting from an error you handle into a speed you run at. If you are managing several providers or several keys, that logic belongs in one place, which is essentially what an AI gateway does.

Reducing the load instead

Every technique below lowers consumption rather than smoothing it, which is the more durable answer.

• Cache aggressively. Identical inputs producing identical outputs should not reach the API twice. Deduplicating at the application layer typically removes more traffic than people expect.

• Cut the context. Most requests carry more history than the task needs. Since tokens are the unit being metered, trimming context reduces TPM pressure directly and cuts your bill at the same time.

• Reuse the fixed prefix. Where the provider supports caching a repeated prompt prefix, long system prompts and static documents stop being re-metered on every call.

• Route by difficulty. Simple extraction and classification does not need your largest model. Sending it to a smaller one frees headroom on the limit that actually constrains you.

Most of these overlap with cutting your API costs, which is not a coincidence: rate limits and bills are two views of the same consumption.

Questions

Why did my limit change without warning?

Tiers move with account age and spend on most platforms, usually upward. Trial and free tiers can also be adjusted during periods of heavy demand. Treat published limits as current values, not guarantees.

Are limits per key or per account?

Usually per account or organisation, sometimes per project. Issuing more keys to get more throughput generally does not work and can breach terms of service. Check the specific provider before designing around it.

Does streaming help?

It improves perceived latency and nothing else. The tokens are still counted, so streaming has no effect on TPM.

What should users see when I hit a limit?

Something honest and specific: the system is busy, the request is queued, here is roughly how long. Silent failure and an indefinite spinner are worse than the delay. The same reasoning applies to running out of credits mid-task, where the failure arrives halfway through work the user thought was progressing. For the underlying mechanics of why token throughput is the binding constraint at all, start with how models actually process a request.