๐๐๐ถ๐น๐ฑ๐ถ๐ป๐ด ๐ ๐ฅ๐ฒ๐๐ถ๐น๐ถ๐ฒ๐ป๐ ๐๐ ๐๐ฎ๐น๐น๐ฏ๐ฎ๐ฐ๐ธ ๐ฆ๐๐๐๐ฒ๐บ
I showed my new project at a meetup. The demo froze. The AI provider was down. My screen showed a loading spinner. I decided to fix this.
One AI API is easy to use. But outages happen. Rate limits hit you. Costs spike. Providers change rules. You get stuck.
I tried a few things:
- Round-robin switching. Half the requests failed.
- Manual code blocks. The code became a mess.
- Third-party tools. I still relied on one company.
I built a Python system. It treats providers as plug-ins. It uses a common interface. It checks health. It rotates through a list of providers.
If one provider fails, the system switches to the next. It happens silently.
There are trade-offs:
- Latency. Retries take time.
- Cost. You might pay two providers.
- Complexity. You manage more keys.
Skip this if:
- You are building a small prototype.
- Your app handles batch jobs.
- You need a specific model.
Do not rely on one pipe. Use a common interface. Start every project with two providers.
How do you handle AI outages?