Our service starts showing problems at 2:15, and someone notices a deployment completed at 2:07. The release immediately becomes the main suspect, even though the real cause may turn out to be an upstream dependency that had a short outage.
We've started adding deployment events to incident timelines, but it's still easy to anchor on the most visible recent change instead of checking what actually changed first. For teams that deploy frequently, what process do you use to separate correlation from causation? Do you rely on rollback, observability data, dependency checks, or a specific incident workflow?
4 Answers
Start with a hypothesis and look for signals that support or contradict it. Check whether the anomaly began during the rollout, whether the affected routes match the code change, and whether unrelated services saw the same symptoms. A deployment that finishes and leaves the system healthy for several minutes may still cause a slow-burn problem such as memory exhaustion, so check resource trends too. Overlaying rollout markers directly on your dashboards is much better than relying on people’s memory of the timeline.
Check dependencies separately before spending an hour reviewing the release. Health dashboards, synthetic checks, traces, and dependency-level latency or error metrics can quickly show whether a vendor or upstream service was already failing. A good incident view should follow the request through the dependency tree and include application deployments, configuration changes, container events, resource failures, and external-service health. That gives you evidence for the most likely impacting component instead of automatically blaming the newest release.
Make every relevant change easy to compare, not just application deployments. Put deployment markers alongside error rate, latency, saturation, and request volume, then add feature-flag changes, configuration pushes, infrastructure events, and dependency incidents to the same timeline. Compare the symptom start with when each change actually took effect. Deployments get blamed so often because they’re usually the only changes teams can see.
That’s probably one of our biggest gaps. We have CI deployments in the timeline, but feature flags and configuration changes are still scattered across different systems. A single change-event stream would make incident review much easier.
Use rollback as an experiment when it’s safe and the impact is clear. If rolling back removes the problem, investigate the release afterward; if the problem remains, you’ve ruled out a major hypothesis and can move on. For intermittent issues or releases with a large blast radius, rely more on timelines, traces, and controlled comparisons. If rollback is consistently too risky or slow, improving that capability is worth treating as a reliability project.
Agreed, although intermittent failures and high-risk rollbacks make this less decisive. In those cases we usually have to compare the timing and error signature against every other change around the incident.
The error signature is especially useful. If unrelated services fail at the same time, or an upstream API is unhealthy, that points away from the release much faster than staring at the diff.