How do teams handle critical cloud resources becoming unavailable when nobody is immediately available to respond? For example, if blob storage in one region fails, would you automatically provision a replacement in another region—or even another cloud provider if the outage affects the whole platform? This discussion is intentionally focused on provisioning and configuration; data replication and consistency are separate challenges.
Some of this can be handled with native monitoring and automation tools, but making changes outside the infrastructure-as-code repository can create drift and multiple sources of truth. One idea is to define condition-and-action rules directly in a Terraform repository, then run a scheduled workflow that evaluates resource health or monitoring queries. When a rule is triggered, the workflow could update Terraform, open a pull request, or—only for tightly controlled cases—automatically merge an approved change so the normal deployment pipeline remains responsible for applying it.
The same approach could potentially adjust resource attributes such as instance size or SKU in response to traffic spikes or prolonged idleness, with recovery conditions that roll changes back when conditions normalize.
Does this solve a real problem, or are native autoscaling and health-recovery features usually sufficient? Which changes, if any, would you trust to apply automatically without review? I'm especially interested in experiences with failover, scaling rules, drift prevention, testing recovery paths, and detecting automation that starts making repeated or inappropriate changes.
5 Answers
In practice, we focused less on recreating a failed resource and more on keeping services useful without it. For storage, that can mean serving cached data, queueing writes, or disabling a feature behind a flag until the dependency returns. A failover plan that has never been exercised is only documentation, so recovery paths should be deliberately tested during normal working hours. Provisioning a replacement is usually easier than deciding when it is safe to switch and proving that the state is correct.
Terraform is probably the wrong layer for immediate resilience. The better approach is to design the application and platform so they already tolerate failures: replicated storage across the availability or regional boundaries you can afford, multiple service replicas, standby systems, DNS or load-balancer failover, and application-level fallback behavior. Cloud-native features can often provide this without Terraform changing anything during an incident. Of course, every extra nine costs money, and your uptime can never exceed the reliability of the services underneath you.
The biggest danger is creating another path that can change infrastructure outside the normal workflow. It is similar to database schema drift: a direct emergency fix is made, but the declared configuration is never updated. If automation is used, it should still commit a change to the repository and let the existing pipeline apply it. The action should never modify real resources directly.
For approval, use blast radius as the dividing line. Cheap, reversible, allowlisted changes could be automatically merged, while expensive, destructive, or difficult-to-reverse changes should require review. Also monitor the automation itself. A rule that is silently wrong and keeps firing may be more dangerous than a single bad change, so every automated operation needs an audit trail, alerts, and limits on repetition.
That is the reason I was leaning toward repository commits rather than direct applies. I had not fully considered how to detect a rule that quietly keeps doing the wrong thing, so logging every automatic operation and alerting on those changes would need to be part of the design.
I would separate recovery from capacity and cost changes. A tested rollback or recovery action can sometimes be allowed to merge automatically, while changing SKUs or scaling up should usually require approval because it affects cost and may have a larger blast radius. Automatic merging could still be safe if the diff must match a strict allowlist and the system leaves a clear record of what happened.
An allowlisted middle ground seems more practical than either leaving every change waiting for someone overnight or pushing all generated changes straight to the main branch.
A failover mechanism can be much more dangerous than an idle-resource rule. If a cost rule is wrong, you may get a slower request or an unnecessary bill, which is usually reversible. If storage failover is wrong, you can end up with conflicting copies of data or an unsafe split-brain situation. I would consider automatic handling for low-risk idle resources only after testing, and keep cross-region or cross-provider failover under manual control until the recovery process is well proven.
That makes sense for customer-facing systems where the cost of high availability is justified. Internal infrastructure often has tighter cost constraints, though, and the scaling or cost-optimization rules are a somewhat separate concern from strict failover.