I’ve been thinking about a failure-handling question that seems simple at first, but gets messy in production:
When an LLM call fails, when should the app retry the same model, and when should it fallback to another model?
For normal API work, retries are often the default instinct. But with LLM apps, the failure can mean very different things:
- temporary provider overload
- request rate limit
- token rate limit
- model-specific capacity issue
- timeout during a long response
- streaming interruption
- quality or formatting drift
The tradeoff I keep running into is that fallback is not “free.” A different model may have different latency, output style, context behavior, tool-calling behavior, or safety defaults. In some workflows, that is acceptable. In others, it can quietly change the product behavior.
Right now, my rough mental model is:
- retry if the failure looks temporary and the operation is safe to repeat
- queue or slow down if it looks like a rate/token limit
- fallback only if the user experience is more important than model consistency
- fail closed if retrying could duplicate a real-world side effect
But I’m not sure where teams usually draw the line.
For people running LLM features in production: do you have a clear policy for retry vs queue vs fallback, or is this usually handled case by case after incidents happen?