A bad Terraform apply recently targeted the wrong workspace and removed a substantial part of a staging-adjacent VPC. The incident exposed a gap in our recovery planning: we had backups for data stores, but not a complete rebuild plan for IAM roles, networking, infrastructure configuration, or cross-service dependencies. It also highlighted the difference between having snapshots and being able to restore an entire working environment. For client accounts, how often do you test full-account recovery, and what cadence gives you confidence without consuming a huge amount of engineering time?
3 Answers
Add guardrails around the IaC workflow as well as recovery testing. Separate workspaces and accounts where possible, require reviews for destructive plans, restrict state access, and use policy checks to catch changes aimed at the wrong environment. Those controls reduce the chance that the recovery plan is needed in the first place.
For most teams, quarterly full-recovery exercises are a reasonable baseline, with smaller restore tests performed more often. The full test should cover identity, networking, service configuration, secrets, dependencies, and data—not only snapshots. Record the time, failures, and manual steps so each exercise improves the runbook.
Treat the account itself as rebuildable infrastructure. Keep Terraform or another IaC system for the environment, document dependencies and required ordering, and use backups for the data. The important part is proving that the code and documentation can produce a usable environment, not just assuming they will.
A strong test is to run the rebuilt environment as the real workload for a while. If nobody is willing to rely on it, that usually means the recovery process still has untested gaps.