I'm looking for guidance on setting up disaster recovery for my client who's having a tough time migrating workloads from the UAE region to Mumbai. They've been down for four days already! We're dealing with 6-7 EC2 instances, 2 ECS clusters, CodePipeline, CodeDeploy, RDS, Auto Scaling Groups, ALB, and S3 without any use of Terraform or CloudFormation. I'm currently trying to copy EC2 and RDS snapshots to the Mumbai region but facing some significant delays and application errors because of the problems in the UAE Availability Zone. What would you recommend as the best migration or recovery strategy in this situation?
1 Answer
I can totally sympathize with your situation; I've been in a similar bind. First off, if your snapshot copies are stuck, cancel them and try copying one at a time using the CLI instead of the console. It might give you better error visibility and could prioritize healthier paths. If you're on Business or Enterprise support, reach out for a Sev-1 ticket; AWS can often escalate those snapshot copies from degraded regions.
Focus on RDS first since it's your critical dependency. Get that snapshot moving to ap-south-1 and set it to restore while you work on other components. Avoid enabling Multi-AZ during your emergency restore because that'll just slow things down—turn it back on once everything is stable.
Once your EC2 snapshots arrive, create AMIs and launch them in a new VPC. It's a good idea to try mirroring your UAE CIDR ranges to avoid a flood of config changes. Re-document your security groups from the UAE console now, before things potentially get worse over there. After your compute and database are up, set up the ALB, register your instances, and use Route 53 for a smooth DNS cutover—keep the UAE records at 0% until you've stabilized for a couple of days.
For ECS and other services, make sure to export your task definitions and pipeline configurations before any major issues hit the UAE region. If there was any cross-region replication with S3, you might find your data has made its way to another region already regardless.
After it’s all sorted, remember to check for any hardcoded region references in your application. Use grepping to catch anything tied to the old region. And with no Terraform or CloudFormation in use, you might want to check out Former2 as it helps in documenting your resources moving forward. Hope you get it resolved soon; four days down is just too long!
Good luck! That's tough to deal with.