Model Fallback Strategy
A backup plan for the AI models your workflows depend on — when the main model goes down or slows down, traffic switches to a second choice automatically instead of everything stopping.
Think of the backup generator at a hospital. When the power flickers, nobody runs around looking for candles — the generator kicks in on its own, the lights stay on, and someone gets a note that it happened. A fallback strategy does the same thing for the AI models behind your workflows.
The problem, in plain words
Your assistant runs on a model from one of the big providers (Anthropic, OpenAI, Google). One afternoon that provider has a bad day: an outage, a rate limit (a cap on how many requests you can send per minute), or answers that suddenly get worse. Every workflow built on that model has the same bad day at the same time. Customers see error messages, your team scrambles, and the fix — point everything somewhere else for a while — is a code change made in a hurry at the worst possible moment. You did nothing wrong, and you are still the one apologizing.
What we set up
Each workflow gets a written backup plan called a fallback chain: a first-choice model, a second choice, and sometimes a third route that is slower but keeps things working. All model requests pass through one gateway (the front door for every AI call), which constantly checks the health of each provider. When a provider times out, hits a rate limit, or starts erroring too often, a circuit breaker (an automatic switch that stops sending traffic to a failing route) flips, and requests flow to the backup. Every switch is recorded in the trace — the run's logbook — so you can measure afterward whether quality dipped while the backup was in charge.
How it works, step by step
- Every request goes through one gate
All model calls pass through the gateway, which is the one place that knows each workflow's backup plan.
- The gate watches for trouble
Timeouts, rate-limit errors, and rising failure rates are health signals it checks continuously. Trouble is noticed in seconds, not when a customer complains.
- Traffic switches to the backup
The circuit breaker flips and requests flow to the second choice. Users keep getting answers; most never notice anything happened.
- The switch is written down
Every fallback event lands in the trace, so you can see exactly which requests ran on the backup and compare their quality.
- Your team is informed, not panicked
The on-call person gets a notification that the switch happened. Nobody is paged to go fix a live outage, because there is no live outage.
- Traffic comes home
When the main provider recovers, the gateway routes requests back to the first choice — also logged.
What changes for you
Before: a provider incident was your incident — visible errors, a scramble, an apology. After: it is a brief detour that your users mostly never see, with a record you can review calmly the next morning. Because every switch is traced, you can also answer the harder question — did answers get worse during the detour? — with data instead of a feeling. What it won't do: the backup model may not be quite as good as your first choice. A fallback keeps you running; it does not promise identical quality — which is exactly why every switch is logged and measured.