I'm a platform engineer at a cloud-resilience company, and we promote Kubernetes recovery as a major part of our product. During a production incident, I discovered that our own infrastructure-as-code configuration had drifted badly from reality. I tried to clean up some old Terraform modules so they matched the current environment, but I made the changes in production first and then treated them as the new source of truth.
When the cluster failed and we triggered our automated recovery process, it faithfully recreated the broken configuration I had just committed. Manual fixes were lost, node pools had the wrong sizing, network policies were missing, and an outdated PostgreSQL version was restored. I'm looking for practical advice on preventing this kind of failure: controlling production changes, detecting drift, recovering lost configuration, and regularly proving that a rebuild from code actually works.
1 Answer
The cleanup itself wasn’t the root problem; the real issue was that recovery assumed the code was accurate without continuously verifying it. In the short term, reconstruct the missing changes from previous state versions, cloud audit logs, Kubernetes backup history, and database backups. Prioritize PostgreSQL, verify the exact version and configuration, and pin it explicitly so recovery cannot silently choose an obsolete release.
For the longer term, use scheduled drift detection, manage Kubernetes resources through a controlled deployment system such as Argo CD or Flux, and require emergency manual changes to be reconciled back into code quickly. Avoid production applies from laptops, and periodically rebuild the environment in an isolated sandbox from IaC alone, then compare it with production. That is the only reliable way to prove the recovery process reproduces the intended state.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures