What's the Best Way to Set Up Kubernetes for Disaster Recovery?

Distributing your control plane across multiple AZs works if those zones have good bandwidth and low latency, plus an odd number of masters for quorum. This setup's pretty common with various cloud providers. Just avoid spreading your control plane across different regions. Keep in mind that the workloads you're managing and their storage needs will impact your decision. Synchronizing between two clusters can be tricky, but some applications, like Harbor, can do some heavy lifting here. You might also consider a global load balancer for easier failover.

A two-cluster setup sounds safer to me. Stretching etcd over unreliable connections is way too complicated. You could use GitOps tools like Argo or Flux to keep both clusters in sync, and for managing containers, replicate Harbor. For databases, rather than going active-active, you might want to set up asynchronous replicas or have a backup and restore plan depending on your RPO and RTO. Velero is excellent for backing up clusters too. Just handle failover at the DNS or load balancer level, keep it straightforward, and make sure to test your cutover regularly!

I have a setup with 7 master nodes: three in one AZ, three in another, and one in an EC2 instance that doesn't run workloads. This way, as long as you don’t have two AZs dropping at once, you shouldn't hit any etcd quorum issues.