When a production system suddenly becomes slow, how do experienced engineers or SREs reason through the investigation when the root cause is unknown? Suppose you have access to logs, recent deployment details, application health, database metrics, external dependency metrics, and infrastructure dashboards. Do you follow a standard sequence, or do you choose the next step based on the evidence available? How do you avoid being misled by an early hypothesis, and what do you do when there is plenty of data but no clear explanation? I'm also curious whether AI-assisted investigation tools are useful in this process and what the biggest practical challenges are.
3 Answers
I wouldn’t automatically walk through every data source in a fixed order. First establish the incident’s timeline and scope: when did latency increase, which endpoints or users are affected, and whether errors, traffic, or saturation changed at the same time. Then check for recent changes, especially a deployment or configuration change. If something shipped shortly before the problem, it becomes a strong lead, not proof of causation.
A practical early split is to compare the application, database, and external dependencies. If database query time or connection-pool usage rose with request latency, investigate there. If calls to one dependency are hanging, use traces and dependency timings to follow that path. If those look normal, then check infrastructure saturation such as CPU, memory, network, disk, or scheduling. Logs are much more useful after you know which service, endpoint, or time window to examine; searching everything at the start often creates noise.
Treat every observation as evidence, not as a conclusion. Build a few competing hypotheses and ask which check would best distinguish between them. For example, a latency increase with normal traffic but only after a release points toward the change, while latency across every service at the same time points more toward shared infrastructure or a dependency. Compare affected and unaffected requests, use distributed traces when available, and keep a timeline of findings. If the evidence conflicts, pause and revisit the scope or measurement rather than forcing the first plausible explanation. AI tools can help correlate dashboards, deployments, and traces or suggest queries, but they still need human validation because they can confidently connect unrelated events.
So the next investigative step is chosen by looking at what changed and which signals moved at the same time, rather than following a rigid checklist?