I'm building a CLI agent that should automatically switch between OpenRouter, Ollama, OpenAI, Anthropic, and Gemini when a provider hits rate limits or becomes unavailable. I wrote a ProviderPool that tracks exhausted providers with timestamps and retries them after a configurable cooldown. It works, but I'm wondering whether there's already a library designed specifically for cross-provider fallback. Most LLM libraries seem to handle retries within one provider rather than switching between several. Has anyone solved this with an existing tool, or is a custom provider pool still the most practical approach?
4 Answers
Using Ollama as a fallback can work well for bulk or less demanding jobs, while cloud models handle quality-sensitive requests. Just account for cold-start latency and model loading time when choosing providers. For a single-process CLI, your own pool is perfectly reasonable; an existing router becomes more valuable once you need shared state, metrics, load balancing, and consistent failure handling.
Your timestamp-based cooldown is a reasonable foundation. I’d also track rolling success rates and latency for each provider. A provider that is technically healthy but consistently slow may be a worse choice than another available provider, and repeated failures should reduce its selection priority over time.
LiteLLM’s router supports fallback between providers, including configurable fallback models. It may be more configuration than a small custom ProviderPool, but it could save you from maintaining the retry and routing plumbing yourself.
If Anthropic is part of the pool, make sure your normalized usage data keeps cache-read and cache-write input tokens separate from ordinary input tokens. They have different pricing, so combining everything into input_tokens can produce inaccurate cost estimates. Also distinguish requests-per-minute from tokens-per-minute limits. A tiny health-check request only tells you about request availability, not whether the token bucket has room. The remaining-token rate-limit header from real calls is more useful for deciding when to restore a provider.
Latency tracking makes sense, especially for Ollama. A local model can take 30 seconds or more on its first request while a cloud provider responds quickly, even though neither one is technically unavailable.