What’s your first five-minute checklist when production goes down?

September 10, 2026 5 answers
Asked by MellowCedar47
Imagine receiving an alert that a production application may be unavailable, but you have no context yet. What sequence do you follow during the first five minutes? I'm especially interested in the order of operations—how you confirm the impact, narrow the scope, communicate, and decide whether to fix or escalate—rather than a list of specific tools.

Answers

OrbitingPine32
A practical sequence is: confirm customer impact, check monitoring and recent alerts, inspect the application and dependency health, review recent deployments or configuration changes, then check DNS, networking, servers, storage, and logs. Send an initial status update early, and escalate or start an incident bridge if the scope is serious.
Reply to this answer
ClearHarbor8
Start by confirming the alert and reproducing the problem from one or two relevant locations. Figure out whether the service is completely unavailable or only partially affected, and whether the issue is limited to certain users, regions, or endpoints. Then check active alerts and monitoring before digging into logs.
Reply to this answer
SilverKite24
The best first five minutes are enabled by good preparation: customer-facing probes, API and dependency checks, cluster and pod monitoring, and infrastructure alerts for CPU, memory, and disk. Alerts should have sensible warning and critical thresholds plus a runbook. Avoid alerting on everything—alert fatigue makes real incidents harder to recognize.
Reply to this answer
QuietMarble61
I usually test from both ends. From the front, verify DNS, routing, load balancers, and the actual application request. From the back, check databases, caches, queues, and connection pools for failures or runaway queries. Those quick checks often identify whether the problem is at the edge, in the app, or in a dependency.
Reply to this answer
AmberVale70
Don’t forget storage. A full disk can cause an application to fail in surprisingly indirect ways, so it’s worth checking early along with system and application logs.

MellowCedar47: That makes sense, though disk usage should ideally already have warning and critical alerts so the incident starts before the filesystem is completely full.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions