Our service recently started having problems at 2:15, and a deployment had completed at 2:07. Everyone immediately focused on the release, but an hour later we found that an upstream dependency had experienced a brief outage. We already include deployment events in incident timelines, but recent changes can still overshadow other possible causes. For teams deploying many times a day, what processes or observability practices help separate correlation from causation?
4 Answers
Check dependencies independently instead of treating your own deployment log as the whole world. Synthetic probes for important third-party APIs, dependency health dashboards, distributed traces, and service maps can quickly show whether an upstream system was failing first. Automated checks before and after deployments are useful too: verify service health, restarts, logs, latency, and key user journeys. The strongest incident reviews explain the timing and causal chain across the service, its dependencies, and infrastructure rather than simply naming the most recent change.
Use a rollback as a controlled experiment when it's safe and the effect should be immediate. If the problem disappears, investigate the release in more detail; if it persists, you have strong evidence to look elsewhere. Rollback isn't always practical for intermittent issues or changes with a large blast radius, so those cases require better timelines and dependency checks. If rollback is consistently too risky or slow, improving that capability is worth treating as separate engineering work.
Start with a hypothesis and look for signals that support or contradict it. Compare the deployment completion and propagation time with the first increase in errors or latency, check whether the affected endpoints match the code change, and look for resource symptoms such as memory growth or saturation. If the system was healthy after the rollout and failed later, a slow-burn issue or an external dependency becomes more plausible. A tight five-to-ten-minute correlation window is a useful initial check, not a substitute for understanding the full failure chain.
Overlaying rollout markers on the dashboards is especially helpful. If the error trend starts before the release has finished reaching all instances, the deployment is much less likely to be the trigger.
The goal isn't really to prove a deployment innocent; it's to make the deployment easy to confirm or eliminate. Put deployment markers directly on error-rate, latency, and traffic graphs, then compare the symptom's actual start time with the rollout timeline. Also include feature-flag changes, configuration pushes, infrastructure events, and vendor incidents in the same timeline. Deployments often get blamed simply because they're the only changes teams can easily see.
That makes sense. We already collect CI deployment events, but feature flags and configuration changes are still blind spots for us. Do you put all of those into one event stream, or rely on an observability tool to correlate them?
Rollback is probably the cleanest test when the risk is low. For intermittent failures or high-impact releases, though, we usually have to compare timelines and account for everything else that changed around the same time.