When production suddenly starts behaving differently, how do experienced on-call teams determine what changed, when it changed, and whether it was intentional? The cause could be a deployment, configuration or feature-flag update, infrastructure event, scaling action, cloud-provider issue, dependency failure, or unexpected drift.
I'm especially interested in the investigation workflow rather than specific tools. Where do you look first, which sources of truth do you trust, how do you connect an alert to a likely change, and which parts still require manual work? I'd also like to hear how teams balance restoring service quickly with finding the deeper root cause, particularly in AWS and Kubernetes environments.
4 Answers
The change-management system should be the primary source of truth, including manual changes, automated pipelines, infrastructure-as-code, and GitOps activity. Ideally every production change is recorded automatically with an exact timestamp and enough context to trace it back to a commit, ticket, or owner. If the timeline is empty, don’t assume nothing changed: compare the actual deployed artifacts. Floating dependency versions, mutable image tags, or a rebuilt image can produce different software from the same commit.
I first establish what is actually broken and what recovery should look like. Then I verify the alert against the system: check whether the affected pod or service is healthy, inspect the relevant logs, and look at traffic, latency, errors, and resource metrics. On hosts, basic system information can still reveal resource exhaustion, OOM kills, kernel errors, disk problems, or other issues that application dashboards miss. After recovery, investigate what created the bad state and how to make diagnosis or prevention faster next time.
The hardest part is usually not the obvious deployment—it’s correlating evidence scattered across several systems. Teams work much better when deploys, config changes, flags, provider events, alerts, and incident decisions can be viewed as one timeline. During the incident, record observations separately from hypotheses and note what was tried or ruled out. Restore service first if necessary, then do a focused review to confirm the root cause and assign follow-up work. Tools can help summarize logs or changes, but they shouldn’t replace direct verification and engineering judgment.
Start with a short timeline around the alert—usually the previous 30 minutes or so. Check deployment and merge history, CI/CD runs, GitOps or Helm activity, configuration and feature-flag events, cloud audit logs, and any approved change records. Cross-reference their timestamps with the alert. In many incidents, that quickly answers what changed and whether it was intentional. Metrics and logs explain the symptom, while the change history often points to the cause.
That makes sense. It sounds like building the timeline first prevents the team from jumping straight into theories.
Configuration and feature-flag updates are a common blind spot because they may not appear in deployment history. They need queryable events on the same timeline as releases and infrastructure changes.