When an important cloud resource goes down and the team is unavailable, how much of the recovery process should be automated? For example, if blob storage in one region becomes unavailable, would you automatically provision a replacement in another region or even another cloud provider? This discussion is focused on provisioning and infrastructure changes rather than data replication.
It seems possible to automate responses with monitoring tools, but making changes directly can cause infrastructure drift from the Terraform repository and create multiple sources of truth. I'm considering a scheduled GitHub Action that evaluates condition-action rules defined alongside the Terraform code. A rule might respond to a resource health failure or a KQL query showing a problem, then modify the Terraform configuration and either open a pull request or, for carefully controlled cases, merge automatically. The normal infrastructure pipeline would remain responsible for applying the change.
The same system could potentially adjust resource attributes such as SKUs or instance sizes in response to traffic spikes, or scale down resources that are consistently idle. Recovery rules could also reverse those changes when conditions return to normal.
I'd like to hear from people with experience operating larger or more complex environments. Are native features such as autoscaling, health alerts, and built-in failover already sufficient? Would you trust automated infrastructure changes without manual review if the allowed changes were explicitly defined? Would you separate low-risk scaling from higher-risk failover, and what problems have you encountered with approaches like this?
5 Answers
For a storage outage, recreating the resource may not be the best recovery strategy. We’ve kept services usable by reading from cache, queueing writes, and disabling nonessential features behind a flag. A failover process that has never been exercised is just documentation, so recovery paths should be tested deliberately during normal hours. Provisioning is usually the easy part; deciding when to switch and handling state consistently are much harder.
Terraform probably shouldn’t be the mechanism that performs an emergency failover. Resilience needs to be designed into the application and platform first: replicated storage, multiple availability zones or regions, redundant service instances, and traffic switching through a load balancer or DNS. The right design depends on the availability target and the cost you’re willing to accept. Native features can usually handle many of these changes without rewriting Terraform.
The biggest danger is creating another path that can change infrastructure outside the declared configuration. The safer pattern is for automation to commit or create a pull request, while the existing pipeline remains the only thing allowed to touch real resources. Otherwise it becomes similar to someone manually fixing a production database and never updating the migrations.
That’s why I’d have the action change the repository rather than apply anything directly. I hadn’t fully considered how to detect a rule behaving incorrectly over time, though. Logging every automated operation and alerting on changes would at least make that behavior visible.
I’d separate the risk levels. Automatic rollback or recovery can be reasonable once it has been thoroughly tested, but changes involving larger SKUs, significant scaling, or cross-cloud failover deserve tighter controls. A useful compromise is to allow automatic merging only for narrowly defined, allowlisted changes while keeping a complete audit trail. That avoids waiting for someone to approve a harmless action at 3 a.m. without allowing arbitrary infrastructure edits.
An unattended pull request doesn’t help much during an incident, and pushing every change directly to the default branch is risky. Allowlisted changes with automatic merging seem like a reasonable middle ground after the behavior has been proven.
Automation is much safer when it reduces waste than when it changes the source of truth for critical data. Automatically shrinking an idle service may only cause a slower request if the rule is wrong, while automatically replacing storage can create conflicting copies or unclear ownership. I’d start with reversible, low-cost changes, measure them carefully, and keep high-impact failover decisions manual until the process has been tested repeatedly.
That makes sense for customer-facing services with strict availability requirements. For internal systems, cost often makes full native redundancy impractical, and the separate ideas around scaling or changing resource attributes may still benefit from controlled automation.