Imagine receiving an alert that a production application may be unavailable, but you have no context yet. What sequence do you follow during the first five minutes? I'm especially interested in the order of operations—how you confirm the impact, narrow the scope, communicate, and decide whether to fix or escalate—rather than a list of specific tools.
5 Answers
Don’t forget storage. A full disk can cause an application to fail in surprisingly indirect ways, so it’s worth checking early along with system and application logs.
I usually test from both ends. From the front, verify DNS, routing, load balancers, and the actual application request. From the back, check databases, caches, queues, and connection pools for failures or runaway queries. Those quick checks often identify whether the problem is at the edge, in the app, or in a dependency.
The best first five minutes are enabled by good preparation: customer-facing probes, API and dependency checks, cluster and pod monitoring, and infrastructure alerts for CPU, memory, and disk. Alerts should have sensible warning and critical thresholds plus a runbook. Avoid alerting on everything—alert fatigue makes real incidents harder to recognize.
Start by confirming the alert and reproducing the problem from one or two relevant locations. Figure out whether the service is completely unavailable or only partially affected, and whether the issue is limited to certain users, regions, or endpoints. Then check active alerts and monitoring before digging into logs.
A practical sequence is: confirm customer impact, check monitoring and recent alerts, inspect the application and dependency health, review recent deployments or configuration changes, then check DNS, networking, servers, storage, and logs. Send an initial status update early, and escalate or start an incident bridge if the scope is serious.
That makes sense, though disk usage should ideally already have warning and critical alerts so the incident starts before the filesystem is completely full.