How do you troubleshoot issues after a production deployment failure?

Usually, the first warning doesn’t come from knowing it’s a deploy issue; it’s more about realizing something is broken and then figuring it out from there. The tools you use can vary significantly depending on your platform. Some setups have a single repository for all changes, while others are more scattered. Sometimes the application performance monitoring doesn't help because the problem might not be related to the application layer. When you encounter this, where do you look first to diagnose the issue?

When things break, I usually dive into the logs first. I check the SHA of the broken service's image, then review the logs of the build that created it. After identifying the suspect commit, I figure out the best way to remedy the situation—whether that’s rolling back, rebuilding, or a different approach altogether. Is this a common strategy among your teams?

I approach it like a doctor's visit: our Site Reliability Engineering (SRE) team first addresses the symptoms and, if needed, we refer the specific issue to developers for deeper troubleshooting. In emergencies, we gather everyone for a war-room meeting to fix things rapidly. Afterward, we conduct a post-mortem to learn from the incident and refine our processes. How effective has this method been for you?

We implement Quality Assurance on staging and also do a verification in production after every deployment. We run canary deployments on smaller environments to catch issues early. If an error slips through monitoring tools like Sentry or Prometheus, we measure its impact on user experience. If it’s severe, we roll back right away, but if not, we work on a hotfix. Do you find it’s easy to assess the situation quickly when something sneaks past, or does it take time to find out what went wrong?

My immediate response is to roll back the changes, observe the telemetry data, and then see if I can replicate the bug in a pre-production environment. This helps clarify the root cause before we attempt any fixes. Does this sound like a good approach?