Multi-AZ and multi-region designs look good on paper, but how do you keep them reliable as configurations and runbooks change? Is infrastructure as code with drift detection useful, or does it become another dashboard?
What actually helps keep AWS workloads resilient in production?
Answers
Drift detection is useful if it leads to a fix, not just a red tile. A standby region can look healthy while security groups or IAM settings no longer match the primary. An emergency manual change is easy to forget once the incident is over.
Keep the infrastructure as code workflow quick enough to use under pressure, and include backporting emergency changes in the incident follow-up. Check that the runbook still points at resources that exist, too.
Reply to this answer
I'd pick a small set of recovery patterns and test them regularly before adding more dashboards. A failover exercise can expose gaps in permissions, networking and runbooks that a diagram won't show.
Include the backups in those tests. Cross-account IAM permissions can change, so check that recovery still works rather than assuming a backup is usable because it exists.
Reply to this answer
Some of this is an operating discipline problem. Establish a minimum account standard and treat drift, confusing procedures and close calls as things to investigate, not just warnings to acknowledge. Otherwise the tooling can tell you something is wrong without anyone owning the fix.
Reply to this answer
Don't overlook the small dependencies. DNS and NAT can undermine a much bigger multi-region design, so keep them simple and observable. The number of availability zones and whether a second region is worth the cost depend on the workload and budget. More infrastructure isn't automatically better resilience.
Reply to this answer
Think about fault isolation and static stability as design goals, not just diagrams to file away. But be realistic about the operational load. Keeping one region consistent can be hard enough, and a second region adds another environment that needs attention and testing.
Reply to this answer
Add your answer
Answers are reviewed before publication.
granola-fueled-debate:
A written incident report can help make that follow-through visible instead of leaving it in someone's head.