I recently faced a situation where my CI/CD pipeline failed to deploy the latest version of my app. It got me thinking about the best troubleshooting approach in cases like this. Do you usually revert to the previous stable version using git reset, or do you prefer to dive into the logs and fix the issue as soon as possible? I tried to roll back first, but that led to a bigger mess with git, which I eventually managed to resolve. I'm curious about how others handle this.
4 Answers
For non-production deployments, we usually take the time to debug and fix the issue instead of rolling back. In production, we weigh the effort involved before deciding whether to fix forward or revert, especially if the deployment has a lengthy change window. It's all about assessing the situation properly before moving on.
The general consensus seems to vary based on the specific situation. In our case, whether to rollback or fix forward is determined collectively by the team. We always consider the impact of the delay, and sometimes it’s simply easier to roll back if the stakes aren’t extremely high. But for other issues, we prefer to fix forward, especially when time is critical.
In our team, if the CI fails to push the Docker image, it's treated as a build failure. We typically avoid git rollbacks due to the number of developers we have across different time zones. Most of the time, we focus on debugging and moving forward instead. It simplifies our workflow and avoids complications in the project history.
Reverting commits forward keeps the history clean and avoids the common chaos of git reset or force pushes—definitely a better way to manage things!
Typically, when we have a failed deployment, we rerun the deploy for the previous release. If there’s no release artifact, we check out the last git tag and deploy from there. For instances where neither is available, you might use `git reset HEAD~1` to check out the previous commit. In my work with medical software, reverting is essential; we avoid rolling forward unless it’s a minor issue.
The problem with rollbacks often comes down to data. While applications can be restored, data integrity is crucial. You need to have scripts in place to manage any data created during the faulty deployment—otherwise, rollback can generate more issues than it solves.
Absolutely, running the previous deployment is a smart move while you sort out your fixes.
That’s a good approach! It’s always a tactical decision based on time and urgency. Sometimes you're left with no choice but to roll back and wait for an appropriate time to deploy again.