I'm a platform engineer at a cloud resilience company, and during a production incident I discovered that our infrastructure-as-code setup was no longer an accurate representation of the environment. I had been cleaning up old Terraform modules to match how our infrastructure looked at the time, but I made the changes in production first and then promoted them as the new source of truth.
When the Kubernetes cluster failed and we triggered our automated recovery process, it rebuilt the same flawed configuration I had just committed. Manual fixes had disappeared, node pools had the wrong sizing, network policies were missing, and an outdated PostgreSQL version was restored. It was especially embarrassing because recovery is a major part of what our company provides.
How do you structure infrastructure as code and recovery processes so that drift and undocumented emergency changes are caught before they become part of a disaster-recovery rebuild?
2 Answers
The cleanup itself wasn’t the root problem; the real issue was that recovery assumed the code already matched production. The manual hotfixes were an undetected time bomb.
For the immediate recovery work, check older Terraform state versions, cloud audit logs, and Kubernetes backup data to reconstruct what was lost. Prioritize PostgreSQL and explicitly pin its version so a recovery process can’t silently restore an outdated release.
Longer term, run scheduled Terraform plans for drift detection, manage Kubernetes resources through a GitOps controller such as Argo CD or Flux, and require emergency changes to be committed back into the repository within a defined window. Avoid applying from personal laptops, and periodically rebuild the environment in a sandbox from code alone, then compare it with production. It’s a painful lesson, but this kind of gap is common and can be turned into a much stronger recovery process.
The biggest fix is to remove direct production write access wherever possible. Infrastructure changes should go through CI or a controlled IaC runner, with reviews and an approval step for production. If that isn’t immediately practical, schedule regular plans or drift checks that compare the declared configuration with the real environment. That gives the team a chance to review and codify emergency changes instead of letting midnight fixes become permanent and invisible.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures