Following the major outages in October 2025, particularly in the US-EAST-1 region, I've been reflecting on our reliance on Multi-AZ configurations. Although our EC2 instances were operational across various availability zones, we were unable to scale or utilize essential API services like IAM and SQS due to a failure in the control plane—specifically, issues with DNS tied to DynamoDB. This experience highlighted a critical gap in our disaster recovery strategies; while our applications were built to be resilient, our operational capabilities were not. Moving forward, I'm pivoting my focus toward enhancing Cross-Region Control Plane Resilience, rather than just having a cold standby in another region. I'm curious about others' automated strategies for handling a potential control plane failure in US-EAST-1. Here are some specific points to consider: 1. Are you employing Multi-Region Serverless setups? Do you utilize tools like Global Tables or have an entirely separate region with its own deployment configurations? 2. What's your approach to DNS Failover? Are you relying solely on Route 53, or do you have an independent DNS provider ready to activate? 3. Do you have automated processes in place that allow you to manage resources across regions while minimizing reliance on primary region services?
5 Answers
We've implemented Route 53 Health Checks that aren't dependent on the control plane; instead, they rely on the data plane. Our failover strategy uses STOP principals, which allows for an automated switch to a secondary system when needed. I prefer automated failover but ensure we have manual checks during failback to keep data consistent before switching back. This setup gives us the flexibility to utilize global tables while managing when we transition back based on our data consistency needs.
At the end of the day, you're always going to face a single point of failure somewhere in your setup. Not all AWS features are universally available; some functionalities remain tied solely to US-EAST-1. Diversifying with a hybrid multi-cloud approach or having some self-hosted components can really help.
This is a classic example of the CAP theorem in action. AWS’s control plane prioritizes strong consistency, but this causes vulnerability during outages. To survive outages, companies should consider multi-region active-active architectures that accept eventual consistency. Yet, many businesses are reluctant to make that compromise until they face the consequences.
Since the outage, we've been shifting our app to an active-passive multi-region configuration, which is quite the task! It’s not just a matter of clicking a checkbox. Even with DynamoDB Global Tables, you need to set up new streams and consumers in the secondary region, which adds workload. One major pain point is Cognito—there’s no real backup plan or replication for it. If we experience an outage, users must re-register in the new region, which can mess up credentials and lead to password reset chaos later. I wish AWS would prioritize solving these kinds of issues instead of just rolling out flashy new features.
Going multi-cloud seems like the best bet if you want to circumvent these issues entirely. I had a client with a multi-cloud architecture to balance outages between AWS and Azure, but during an incident, we struggled with spinning up containers on Azure because Docker Hub went down due to an AWS outage. So even that has its challenges!
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures