I'm consulting for a client who wants redundancy in their Kubernetes setup. They have isolated k3s clusters with ArgoCD in each, and they want a failover system in case one cluster goes down. The plan involves promoting a standby database server to the primary one through WAL replication and updating DNS to redirect traffic. I have a couple of questions regarding this approach. First, is this failover setup a wise choice when considering Kubernetes' native redundancy, or are there better alternatives? Second, if we decide to go ahead with this setup, I need to manage Git files across both clusters. I'd like to maintain a "main" branch and separate branches for each cluster, synchronizing changes through a CI pipeline. Is this a practical strategy? I'm curious if anyone has insights, especially since my experience has been mostly with smaller setups rather than larger enterprise scenarios.
5 Answers
Have you thought about running both clusters in a high-availability (HA) configuration instead? ArgoCD can handle that pretty smoothly, and HA might be more reliable than a failover setup. Just a thought!
If each cluster has over three nodes, consolidating into a single cluster could be better. Also, for your CI/CD process, consider using an application set with a generator to avoid managing multiple branches. Keeping everything in one repo might streamline updates.
Typically, you handle failover at the application level rather than the cluster. If your use case involves a highly available database, consider deploying that across multiple nodes in a single cluster. If you need to span regions, then managing two separate clusters makes sense, but you'll want to set up your application for redundancy across those instances rather than relying on cluster-level failover.
You might want to explore some documented architectures for HA setups with Kubernetes. Check out the recommended architectures for PostgreSQL and other resources that cover multi-cluster designs. They provide insights that could enhance your failover strategy and offer alternatives you might not have considered. Reading them would give you a solid grounding to build from!
That first article really does have some great insights, especially if you’re new to database management in Kubernetes!
What’s the database situation? Depending on the type, you might find that an active/active setup could simplify things. Just curious about how you're planning to handle database redundancy in this case!
But if you’re looking at two different data centers, you might end up with quorum issues for your etcd. A solid plan could mean running in three zones to mitigate risks.