A couple of our internal services occasionally time out under load. They retry with backoff, usually recover, and users generally don't notice. The problem is that this has been happening for so long that everyone treats it as background noise, and incident reviews rarely question it anymore. How do you decide when timeouts have crossed the line from an expected distributed-systems failure mode to a problem that deserves investigation?
4 Answers
A timeout shouldn’t automatically trigger a major incident, but it should be observable from the first occurrence. Define what normal means for each service and endpoint, then alert on meaningful changes rather than every isolated event. If timeouts are frequent enough to affect capacity planning, customer latency, cost, or the error budget, they’re no longer just harmless noise and should get a proper investigation.
The line is usually crossed when the pattern is increasing, concentrated around a particular dependency or load condition, or no one can explain why it happens. Watch p95 and p99 latency, queue depth, connection-pool exhaustion, garbage-collection pauses, database locks, rate limits, and deploy timing. A stable, understood rate might be acceptable; a slow upward trend is a warning that the system is getting closer to a failure cliff.
Timeouts are inevitable, but “it retries” isn’t enough to decide whether they’re acceptable. Track the timeout rate, retry rate, retry success rate, latency added by retries, and the percentage of requests that still fail after retrying. Put an SLO around both user-visible success and internal retry behavior. A retry can hide a reliability problem until traffic increases, so it’s worth having an error budget and watching its burn rate instead of arguing over whether an absolute number feels high.
Make sure retries are bounded and safe: use a per-request deadline, exponential backoff with jitter, a maximum attempt count, cancellation when the original request is gone, and idempotency protection for writes. Also account for retry amplification. If one user request turns into several downstream calls, a partial slowdown can become self-inflicted overload. Even successful retries matter if they regularly push requests beyond the user experience or latency budget.
That makes sense. We already use backoff, but we’ve mostly treated retries as either present or absent. Measuring their rate and the extra latency should give us a much better signal.