When your LLM provider goes down, what should your app do?

Give every model call a timeout, retry only the errors that clear on their own and cap the retries, and decide ahead of time what the feature does when the model is unavailable: queue the work, fall back, or show a clear message. Keep the API key on the server, behind a spend cap.
- Every production model call needs a timeout sized to the job. Ours run from 15 seconds for a one-line classification to 240 seconds for a batch of social posts.
- Retry overloaded and 5xx responses a small, fixed number of times. Don't retry a 400 or a 401; it will fail the same way again.
- When a feature fails, the user should get a sentence that says what happened and when to try again.
- API keys belong in server-side secrets with a spend cap your own code enforces, because a leaked key is billed to you.
Want this built into an AI feature you already run? Start a project.
Where this comes from: we read the model-calling code in four of our own products on November 17, 2026: ConceptSEO, SocialBotify, P360X and CarSnipe. All four send their language-model calls to Anthropic's API. Every timeout, retry count and message below is copied from that code. Where we don't do something, the page says so.
What breaks when a model API goes down?
Every feature that waits on the model, and usually in an ugly way. On November 13, 2026, Anthropic had two incidents. A sign-in failure ran from 09:58 to 11:58 UTC and hit claude.ai, Claude Code and the Console. Then elevated errors on claude.ai, the Claude API and Claude Code ran from 13:09 to 15:38 UTC, according to the timelines Data Studios and AI TLDR compiled from Anthropic's status page. Only the afternoon incident reached the API, and that is the one a production app feels.
An outage rarely arrives as a clean "down". It shows up as calls that hang, calls that return an overloaded error, and calls that succeed but come back cut short. Each one needs its own handling.
| Failure | What the app sees | What handles it |
|---|---|---|
| Slow or hung call | A request that never returns, and a web worker tied up while it waits | A timeout sized to the job |
| Overloaded or 5xx | HTTP 529, 500, 502 or 503 | A small, capped number of retries with a pause between them |
| Rate limited | HTTP 429 | Back off, then queue the work or tell the user to wait |
| Bad or revoked key | HTTP 401 | No retry. Alert whoever owns the key |
| Partial or unparseable output | A 200 with JSON that won't parse or fewer items than asked for | Validate the output, then retry only what's missing |
| Outage longer than your retries | Every attempt fails for an hour or more | Queue the job for later and show a status message |
The status codes are the ones in Anthropic's API errors reference, where 529 means the API is temporarily overloaded. Other providers use different codes for the same states, so map them once, in one place.
Timeouts, retries and when to stop retrying
Size the timeout to the work. An HTTP client's default timeout was not chosen for a model that may spend two minutes writing. Ours, by job:
| Product | Call | Timeout | Retries |
|---|---|---|---|
| CarSnipe | Read a cancellation note | 15s | None |
| ConceptSEO | Detect a site's business type | 30s | One after 1s, on 500, 502, 503, 529 |
| ConceptSEO | Check a post image | 60s | None; failure means reject |
| ConceptSEO | Run a full analysis | 90s; 180s for long outputs | None; marked failed, run moves on |
| SocialBotify | Write a batch of posts | 240s | One batch retry after 10s |
The retry rule we kept arriving at is narrow on purpose. Retry the errors that clear on their own: overloaded, 500, 502, 503. Don't retry a 400, because the request itself is wrong, or a 401, because the key is. In ConceptSEO's business-type detector that rule is a short list of status codes, and anything outside it is logged and dropped on the first failure.
Cap the retries, and count them in time rather than just in attempts. Two tries a second apart get you past a blip. Five tries with growing backoff during a two-hour outage just hold a worker for minutes and then fail anyway. When the provider is really down, the retry that helps is the one an hour later, and that is a queue's job, not a loop's.
Don't stack retries. Official SDKs ship with retries of their own, so a retry loop wrapped around an SDK call multiplies them. SocialBotify makes one attempt per API call on purpose, with the comment "No API-level retry (batch-level retry provides second chance)". Pick one layer and turn the others off.
Falling back to a second model or provider
None of our four products switches its language-model calls to a second provider, and that was a choice. A prompt tuned against one model family is a different prompt on another. The structured output may not parse, the tone of customer-facing copy changes, and the safety behavior differs. A fallback that hasn't been tested can quietly publish worse output than an honest failure would.
The one fallback we run is inside a single provider. ConceptSEO's business-type detector asks the API for the newest small model, caches that answer for a day, and falls back to a pinned, known-good version if the lookup fails. If the newest model returns JSON the code can't use, it tries once more on the pinned one. The job is the same and the prompt was tested on both, so the fallback is boring, which is what you want from one.
If you do add a second provider, treat it as a second product:
- Keep a prompt per provider and run the same test inputs through both before you rely on it.
- Validate the fallback's output against the same schema as the primary's, and fail if it doesn't fit.
- Record which model produced each result, so a quality complaint can be traced to the fallback.
- Decide per feature. A classification can fall back cheaply. Copy a customer will publish under their name may be better waiting an hour.
Queueing work and telling users what is happening
Anything a user isn't actively waiting on should run in a background job, where a failure costs a retry instead of an error page. ConceptSEO's setup jobs (business type, brand detection, site discovery) each run up to twice with ten minutes between tries. The weekly analysis run marks a failed analysis as failed and moves on to the next one, so one bad call doesn't sink the other types for that site.
For anything a user is waiting on, write the message before the outage, not during it. ConceptSEO maps a 429 to "AI rate limit reached. Please wait a minute and try again." and a 529 to "AI service is overloaded. Please try again in a few minutes." P360X's coach says "Coach couldn't answer. Try again." Each one tells the reader it isn't their fault and what to do next, which a spinner never does.
Then decide what a failure should default to, feature by feature, and lean toward doing nothing visible:
- CarSnipe reads a cancellation note to decide whether to ask the customer for a review. On any failure it sends no ask. The docstring puts it plainly: "no ask beats a wrong one."
- ConceptSEO's image check, which decides whether a photo may go out with a social post, counts a failed check as a rejection. An outage holds posts back; it never waves them through.
- When SocialBotify fails a batch twice, it sends the operator a Telegram alert instead of leaving the customer with an empty screen and nothing in anyone's inbox.
Someone has to hear about failures that are not one-offs. One gap we found while writing this: ConceptSEO’s analysis screen sends a Telegram message when an analysis succeeds and nothing when the API call fails, so a long outage would show up only as missing results. That is the next thing on our list.
Keeping API keys from becoming someone else's bill
Stolen model keys are a market now. In October 2026, Help Net Security reported on Bitsight's research into MONST3R-AI, a criminal tool sold as an "autonomous penetration testing platform". Buyers paste in stolen OpenAI, Anthropic and Google keys and the key owners pay for the scans. In September, a SANS Internet Storm Center diary described an attacker using a coding agent to break into poorly secured LLM resale gateways and resell the access it collected.
What we do about it:
- The keys live in server environment config or, for P360X, in a Cloudflare Worker secret. The browser never sees one. If the secret is missing, P360X's AI endpoint returns a 501 rather than trying anything else.
- Spend is capped in our own code. P360X has a hard cap of $5 a month, tracked per call in a database table. Past it, the API returns a 429 with "Coach has used this month's $5.00 limit. It resets on the 1st." ConceptSEO checks a monthly budget per user before every analysis and refuses with a 422 once it's reached.
- ConceptSEO writes the tokens and cost of every call, prompt-cache pricing included, to a usage table, so a spike shows up as a number against a model, a job and a user.
What we don't have yet, and you should: scheduled key rotation, and spend alerts at the provider as a second layer behind the app's own cap. An app-level cap protects you from your own bugs. It does nothing for a key someone copied out of a log or a laptop, because a stolen key calls the provider directly and never touches your code. Help Net Security's advice is the right bar: treat a leaked model key as seriously as a leaked cloud credential.
A failover checklist
- Every model call has an explicit timeout, sized to the job, not the HTTP client's default.
- Retries cover overloaded and 5xx errors only, are capped, and live in one layer rather than on top of the SDK's own.
- Output is validated against a schema, and a partial result retries only what is missing.
- Work nobody is waiting on runs in a queue with its own tries and backoff.
- Each user-facing failure has a written message that says what happened and when to try again.
- Each feature has a decided default for failure, and for anything that publishes or sends, the default is to do nothing.
- Any fallback model has been tested with the same prompt and inputs as the primary.
- Someone is alerted when failures repeat, not only when things succeed.
- Keys are server-side, scoped per app, rotated on a schedule, and backed by a spend cap in code plus an alert at the provider.
If you'd rather have this built into an AI feature already in production, it's the kind of work our AI web app service covers.
Methodology, and reusing these figures
Every timeout, retry count, budget and message on this page was read from the source code of ConceptSEO, SocialBotify, P360X and CarSnipe on November 17, 2026. Where a product relies on its SDK's default timeout or retries, the page doesn't quote a figure for it. The November 13 incident times come from two independent timelines compiled from Anthropic's status page, linked above. The key-theft reporting is from Help Net Security (October 16, 2026) and the SANS Internet Storm Center (September 11, 2026). This page quotes no prices beyond our own $5 cap.
One limit worth stating: all four products send their language-model calls to a single provider, so this is one shop's experience with one API. SocialBotify also calls OpenAI to generate images, which this page doesn't cover. The retry rules carry over to other providers. The exact status codes don't.
You may republish these figures with attribution and a link to https://concept211.com/articles/llm-provider-outage-failover/. Every heading and table row above carries a stable id, so you can link to a single figure.
Photo by Tyler on Unsplash. Illustrative, and not one of our servers.
What does your AI feature do when the model doesn't answer?
Timeouts, retries, queues, failure messages and spend caps are the unglamorous half of an AI feature, and they're the half that decides what your users see on a day like November 13.