I work at a fast-moving startup and have been asked to improve alerting and monitoring for our critical P0 infrastructure. The goal is to help engineers quickly determine whether an incident is caused by their service or by a lower-level dependency.
For example, if our DNS provider goes down, a product engineer might get paged and spend time investigating a wave of 404s in their service, even though the real problem is the infrastructure underneath it. I'm considering a common triage approach where engineers first check the health of core infrastructure and dependencies, then investigate their own service if those checks look healthy. I'd also like to use automation to assist with this process and measure how much firefighting time is spent on infrastructure issues versus service-level problems.
I'm new to this area and would appreciate advice from teams that have tried something similar. What does your incident-triage process look like, and what worked or failed in practice?
4 Answers
We abandoned the multi-level protocol because nobody reads a flowchart during an overnight incident. What worked better was one dashboard link in the alert with four quick facts: dependency status, the last 15 minutes of error rate, the latest deployment identifier, and the owner. We also limited the initial investigation to a couple of questions before escalating. That reduced the time to find the correct owner from roughly 20 minutes to less than five.
Another useful dimension was whether the issue affected every request or only a subset, such as requests handled by recently restarted workers. That distinction can reveal process-startup or infrastructure problems that average latency and application logs may hide.
Alert correlation is probably more valuable than a human-driven protocol. If a dependency check is red when a service alert fires, the paging system can label the service incident as likely downstream of that dependency. Include a metric for time to correctly identify whether the problem is infrastructure or the service itself, separate from total time to resolution. Total resolution time alone can hide the fact that responders are still starting with the wrong team.
Keep a lightweight central collection of runbooks and ownership links. Every alert should point to a relevant page, even if the runbook starts out nearly empty. As incidents happen, improve the documentation with the commands, checks, escalation paths, and communication steps that were actually useful.
The documentation should be easy to maintain rather than exhaustive. A page at 3 a.m. that links to the right workflow or team can reduce panic and prevent people from escalating to everyone at once.
Don’t make people follow a long checklist while they’re half-awake at 3 a.m. Put the important context directly in the page: dependency health, recent error rate, latest deployment, and the owning team. If a core dependency is already failing, the alert should clearly say the service problem is probably downstream of it.
That’s the balance I’m trying to find. A flowchart probably won’t get followed under pressure, so I’m leaning toward putting the most useful dependency signals directly in the alert and measuring whether engineers reach the correct owner faster.
We have lean runbooks already, but I’m trying to understand how to tell whether they’re providing a real benefit. I’m considering tracking things like time to the first correct owner, time spent investigating the wrong layer, and how often responders use the linked checks.