What to Do When Your AI Provider Is Down
Confirm it is them, not you. Then fall back to the tier you chose in advance, because during an incident you will pick whatever is fastest to type.
The first thing to do when your AI provider is down is confirm it is actually them. Check the vendor status page (OpenAI, Anthropic, or whoever you route through), then check whether your errors are 5xx or 429. A 429 is not an outage, it is a rate limit, and the fix is entirely different. Only after that do you touch anything.
This is a narrower problem than your app breaking in production, and it deserves its own plan, because the failure is outside your code and no amount of debugging your own stack will move it.
What to do when your AI provider is down, in five minutes
In order, because the order saves time:
Status page and vendor status feed. If the incident is posted, you are done diagnosing. Skip to degradation.
Error code. 429 means throttling, 5xx means their side, 401 means your key or billing, timeouts with no code often mean a network path problem rather than the model.
One request from somewhere else. A curl from your laptop against the same endpoint separates "the provider is down" from "our production egress is broken." These look identical in your logs and are not the same problem.
Blast radius. Which features are actually dead? Usually fewer than the dashboard suggests, because most apps have one loud AI feature and several quiet ones.
Tell people before they tell you. A status note costs two minutes and buys enormous goodwill compared to silence.
If the status page is green and a laptop request also fails, the problem is yours. If the laptop request works and production does not, look at your egress, your key, and your spend limits before blaming anyone.
Decide the tier before the outage, not during it
Every AI feature deserves exactly one of four degradation behaviours. Deciding this in advance is the whole trick, because during an incident you will pick whichever is fastest to type.
Tier | Behaviour when the provider is down | Use when |
|---|---|---|
1. Serve stale | Return a cached or previously generated result | The output changes slowly and being slightly old is fine |
2. Queue and retry | Accept the request, process it later, tell the user | The user does not need the answer this second |
3. Fail over | Route to a second provider or a smaller model | The feature is load bearing and the quality drop is acceptable |
4. Honest error | Disable the feature and say so plainly | Anything where a wrong answer is worse than no answer |
The decision rule is one question: what does a bad answer cost here compared to no answer? If a wrong answer is embarrassing but cheap, fail over. If a wrong answer goes into a contract, an invoice, a medical note, or an email to a customer, take tier 4 and mean it. The failure mode nobody plans for is a fallback model quietly producing worse output for six hours while everyone assumes the system is fine.
Write the tier next to each feature in your own notes. It takes ten minutes and it is the single highest-leverage thing in this post.
Building failover that is worth having
Fail over is the tier people reach for first and get wrong most often, because a second provider is not a drop-in.
Prompts are not portable by default. A prompt tuned against one model's behaviour will not produce the same shape of output on another. If your code parses the response, that is where it breaks. The fix is to constrain output structure explicitly and validate it on the way out, so a fallback that formats differently fails loudly rather than silently corrupting a downstream step.
Test the fallback path on a schedule. An untested failover is a second outage waiting behind the first. Route one percent of traffic through it, or run it nightly against a fixed set of inputs and diff the results. If you have never seen the fallback produce output, you do not have a fallback.
Cap the retry storm. When a provider degrades rather than dies, every request slows down, your workers pile up, and your own service falls over from queue pressure while the provider is technically still serving. Set aggressive timeouts, use a circuit breaker that trips after a threshold of failures, and let it stay tripped for a cool-off period rather than hammering the recovery.
Know your rate limits on the fallback. The provider you fail over to has never seen your full traffic. Moving 100 percent of load onto an account provisioned for spare capacity turns a partial outage into a total one. This is where understanding AI API rate limits stops being theory.
What to tell users
Short, specific, no hedging. Name the feature, not the vendor, and give a shape of time rather than a promise.
Draft generation is unavailable right now because of an upstream provider issue. Everything else is working normally. We will update this notice within the hour.
Three things that message gets right: it scopes the damage so people do not assume the whole product is broken, it does not blame a named third party, and it commits to an update rather than to a fix time you cannot control.
Resist the urge to say "brief interruption" before you know. Outages that were called brief and were not are how trust gets spent.
Keeping a second provider without doubling your work
The reason most teams do not have failover is not the routing code, which is trivial. It is the fear of maintaining two sets of prompts forever. That fear is justified if you write prompts inline at every call site, and largely avoidable if you do not.
The pattern that keeps the cost down has three parts.
One place where prompts live. A module, a directory of text files, a table, anything with a single owner. If a prompt appears as a string literal in three request handlers, you now have three places to update per provider, and failover maintenance becomes a real tax.
A validated output contract. Decide what a valid response looks like, structurally, and check it on the way out regardless of who produced it. A schema check plus a retry on failure means a fallback model that formats differently produces a visible error rather than a silently malformed record three steps downstream. This is worth having even with one provider.
A fixed evaluation set. Twenty or thirty real inputs with known-good outputs, stored in the repository. When you add or change a provider, run them and read the diff. Thirty minutes of setup, and it converts "does the fallback still work" from an anxiety into a command.
With those three in place, adding a second provider is a day of work and a small ongoing cost. Without them it is a project everyone postpones, which is why so many outage plans consist of an apologetic status page.
After it clears
Two follow-ups, both quick.
Drain the queue carefully. If you queued work at tier 2, releasing it all at once against a provider that has just recovered is a good way to trip their rate limits and start again. Ramp it.
Write down what you learned about the boundary. Not a formal postmortem, just: which features actually broke, which tier they should have been in, and what you had to look up. The next outage is easier entirely because of notes like that.
The uncomfortable structural point
If a single provider going down takes your product with it, you have a business dependency, not a technical one, and no amount of retry logic changes that. Multi-provider support has a real cost in prompt maintenance and evaluation work, so it is a legitimate choice to accept the risk. Just make it a choice you made rather than one you discovered, and revisit it when the feature it protects starts carrying revenue. Keeping your prompt layer provider-agnostic from the start makes migrating from one AI model to another a routine task rather than a project.
FAQ
How do I tell a rate limit apart from an outage?
The status code. A 429 is throttling and means you are asking for more than your account allows, which is a capacity or budgeting problem. A 500, 502 or 503 is the provider failing. Retrying hard on a 429 makes it worse.
Should I always have a backup provider?
No. It is worth it for features that carry revenue or that users would churn over. For a nice-to-have summarisation button, an honest error message is a perfectly good answer and costs nothing to maintain.
Will caching hide most outages?
It hides more than people expect. A cache keyed on the input, holding results for hours rather than minutes, turns a large share of short incidents into something nobody notices. It also cuts your bill, which is a useful side effect worth reading about in reducing AI API costs.
How long should my timeout be?
Shorter than you think, and different per feature. An interactive feature that has not responded in 15 seconds has already failed from the user's point of view. Background work can wait minutes. One global timeout for both is how a slow provider takes down a fast feature.
Start from the foundations in building an app with AI if you are still choosing how your stack fits together.
How did this land?
About the author
Staff Engineer, Platform
Carlo works on the platform that turns prompts into running apps. He writes the engineering deep dives and the changelog notes worth reading.
