What’s the hardest part of proving a production change caused an incident?

Finding a correlation is only the beginning. If a deployment happened right before an outage, you still need to explain the mechanism connecting that change to the failure. Understand what changed, how the affected components work, and how they interact. A pair of matching graphs isn’t enough to establish causation.

The cleanest confirmation is usually an isolated rollback: revert only the suspected change and see whether the symptom consistently disappears. That isn’t always possible with schema migrations, cache warmups, or other irreversible changes. In those cases, you have to reason from the affected components, blast radius, timing, and whether the failure mode makes technical sense.

Reproducing the behavior in a lower environment can help, but it’s often slower and less conclusive than people expect. Production capacity, traffic patterns, data volume, caches, and dependencies may be different. Treat reproduction as supporting evidence rather than definitive proof, and combine it with timeline data, metric baselines, logs, and an explanation of the failure path.

One of the trickiest distinctions is whether a change caused the problem or merely exposed an existing weakness. For example, a deployment might quadruple database connections and trigger an outage, while the database had already been close to its connection limit for weeks. Reverting the deployment may restore service temporarily without fixing the underlying risk.

A useful approach is to establish the change window from infrastructure and audit history, then compare the relevant metric with the same weekday from one or two weeks earlier. That gives you a better baseline than only comparing it with the previous hour.