How do you tell whether a deployment actually caused an incident?

November 16, 2026 5 answers
Asked by 8m+velvetspoke_624

When a service starts having trouble shortly after a deployment, the release tends to become the main suspect. How do you separate a real cause from a timing coincidence, especially when deployments happen frequently?

Answers

6u!corkwicket_382

Make a timeline that includes more than deployment start times. Overlay rollout events on error rate and latency graphs, then add config changes, feature flags, and relevant upstream events. Compare when the symptoms began with when each change actually reached the affected traffic.

Look at the error signatures and affected routes too. Do they fit what changed? A mismatch is useful evidence, though it does not completely clear the release. Problems can show up after a delay, so check resource trends as well.

8m+velvetspoke_624:

We already capture deployment events, but config and feature flag changes would make the timeline a lot more useful.

Reply to this answer
p.731+foldcrank

If the rollback is safe and quick, do it and watch the same error and latency signals. Recovery after rollback makes the release a stronger suspect. If the problem continues, look elsewhere.

Neither result is absolute proof. An intermittent issue or an upstream recovery can coincide with the rollback, and some changes may not be fully reversed by it.

w.518+buttonchisel:

The tradeoff gets trickier when rollback has a large impact of its own. In that case the timeline and other signals may be the safer first step.

Reply to this answer
m.307~copperstaple

Check the dependencies separately. Distributed traces can show where requests are slowing down or failing, and synthetic probes against third-party APIs can give you a signal outside the affected service.

An upstream failure around the same time does not automatically clear the deployment, but it gives you another concrete hypothesis to test.

Reply to this answer
_a52+drumstencil.7

Build repeatable checks around releases. Compare service health, restarts, logs, and key request metrics before and after rollout. If a failure mode has no useful log or metric, add coverage for it so the next investigation has better evidence.

Reply to this answer
4g!corkratchet_827

A recent change is a reasonable place to start, not a verdict. Frame it as a hypothesis and look for evidence that supports or contradicts it. That keeps the investigation focused on restoring service instead of arguing over whose change is to blame.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions