When a service starts having trouble shortly after a deployment, the release tends to become the main suspect. How do you separate a real cause from a timing coincidence, especially when deployments happen frequently?
How do you tell whether a deployment actually caused an incident?
Answers
Make a timeline that includes more than deployment start times. Overlay rollout events on error rate and latency graphs, then add config changes, feature flags, and relevant upstream events. Compare when the symptoms began with when each change actually reached the affected traffic.
Look at the error signatures and affected routes too. Do they fit what changed? A mismatch is useful evidence, though it does not completely clear the release. Problems can show up after a delay, so check resource trends as well.
Reply to this answer
If the rollback is safe and quick, do it and watch the same error and latency signals. Recovery after rollback makes the release a stronger suspect. If the problem continues, look elsewhere.
Neither result is absolute proof. An intermittent issue or an upstream recovery can coincide with the rollback, and some changes may not be fully reversed by it.
w.518+buttonchisel:
The tradeoff gets trickier when rollback has a large impact of its own. In that case the timeline and other signals may be the safer first step.
Reply to this answer
Check the dependencies separately. Distributed traces can show where requests are slowing down or failing, and synthetic probes against third-party APIs can give you a signal outside the affected service.
An upstream failure around the same time does not automatically clear the deployment, but it gives you another concrete hypothesis to test.
Reply to this answer
Build repeatable checks around releases. Compare service health, restarts, logs, and key request metrics before and after rollout. If a failure mode has no useful log or metric, add coverage for it so the next investigation has better evidence.
Reply to this answer
A recent change is a reasonable place to start, not a verdict. Frame it as a hypothesis and look for evidence that supports or contradicts it. That keeps the investigation focused on restoring service instead of arguing over whose change is to blame.
Reply to this answer
Add your answer
Answers are reviewed before publication.
8m+velvetspoke_624:
We already capture deployment events, but config and feature flag changes would make the timeline a lot more useful.