I work at a fast-moving startup and have been asked to improve alerting and monitoring for critical P0 infrastructure. The goal is to help engineers quickly determine whether an incident is caused by their own service or by a dependency lower in the stack. For example, if our DNS provider is unavailable, a product engineer might be paged for a spike in 404s and waste time debugging their application when the real issue is shared infrastructure.
I'm considering a standard triage approach where responders first check the health of core infrastructure and dependencies, then investigate the affected service if no underlying issue is visible. I'd also like to add automated assistance and collect metrics on how much incident time is spent investigating infrastructure versus service-level problems.
I'm new to this area and would appreciate advice on how other teams handle incident triage. Have you tried a similar process, and what worked or failed? I'm especially interested in practical alert design, runbooks, dependency correlation, and ways to measure whether these improvements actually reduce wasted investigation time.
4 Answers
A compact alert payload can beat a multi-level protocol. One team reduced initial investigation time by putting four items in every page: dependency status, the last 15 minutes of error rate, the latest deployment identifier, and the owner. Another useful question is whether the problem affects every request or only a slice, such as requests handled by recently restarted workers. That distinction can reveal process-age or warm-up problems instead of sending people down the wrong application-debugging path.
Also check whether requests are reaching the application at all. An empty 500 response with no corresponding application logs may indicate an exhausted or empty worker pool, a proxy problem, or another layer below the service.
Measure time to identify the correct failure domain separately from total time to resolve. Resolution time alone can hide misrouting: the incident might finish faster while responders still spend ten minutes investigating the wrong team or layer first. Useful measurements include time to first correct owner, number of escalations before ownership is clear, infrastructure-versus-service classification accuracy, and how often responders use or update the runbook. Review these after incidents instead of treating the original process as permanent.
Keep a lightweight central documentation hub with links to service ownership, access information, escalation paths, and runbooks. Every alert should point to the most relevant page. Runbooks don’t need to be perfect before they’re useful; start with a short checklist or even an empty page and improve it after each incident. The important part is keeping them small enough that teams will actually maintain them. A clear page and escalation path also reduces the instinct to alert every possible team at once.
Avoid making responders follow a long checklist or flowchart during a 3 a.m. page. Put the most important context directly in the alert: dependency health, recent error rate, the last deployment, and the owning team. If a core dependency is already failing, the alert should say that the service may be downstream of it. Automating that correlation is much more reliable than expecting someone under stress to work through several levels manually.
That balance is exactly what I’m trying to figure out. People probably won’t follow a detailed flowchart in the middle of the night, so I’m leaning toward putting the key infrastructure status in the alert and using responder feedback to measure whether it helps.
We’re taking a similar approach with concise runbooks. I’m still looking for a good way to tell whether the documentation and monitoring are producing a real improvement rather than simply adding more links.