I'm running a K3s Kubernetes cluster with only 2 nodes. I realize this isn't an ideal setup for high availability (HA) since true HA typically requires at least 3 nodes to ensure quorum, but my budget constraints mean I can't add a third node. Here's how I'm managing things:
- I have DaemonSets for my frontend, backend, and nginx, with one instance per node. If one node fails, users can still access the app from the other node, so it's somewhat functional for now.
The tricky part is my PostgreSQL setup. I want to run PostgreSQL 16.4 in an active-active configuration across both nodes so that:
- The application and database continue to function even if one node goes down.
- When the downed node returns, the PostgreSQL instances can resync automatically.
I have a few questions about this scenario:
1. Is it possible to achieve this with only two nodes?
2. Is an active-active PostgreSQL setup advisable in K3s?
3. What failure modes should I be aware of, like split-brain or issues with persistent volume claims (PVCs)?
4. Should I consider using solutions such as Patroni, Stolon, or PostgreSQL BDR?
5. Or could I use an external ETCD like kine to mimic a 3-node control plane?
7 Answers
Consider running PostgreSQL separately as a managed service; that way, you treat it like a typical sysadmin task rather than trying to force it into your current infrastructure, which might not be the best design.
Look into Cloud Native PostgreSQL for a 2-node setup; it worked well for me in a past project. That said, watch out for how your cluster behaves if one master node goes down. You'll face challenges with reconfigurations in that case.
You could try using an external datastore, but with the default etcd, you really need 3 nodes to handle failures properly. It's a tough spot with your constraints.
Yeah, not having 3 nodes puts you in a bind. It’s tricky to work around that! Good luck.
It's crucial to know that while your apps might still run if the Kubernetes API server is down, dynamic changes won’t stick until it comes back online. Keep it simple with one master and one worker node, and ensure you’re regularly backing up etcd. If the master fails beyond recovery, you can use those backups on the worker to reset and promote it to master.
Totally agree! Focusing on disaster recovery over high availability is key in your situation. Your nodes are limited, so prioritize safe backups and recovery instead.
You might need to rethink your ingress setup too, as the control plane going down will stop proxying traffic. Make sure both node IPs are in your A record to keep things running!
Active-active PostgreSQL on two nodes is risky. Tools like Patroni or Stolon might help with failover management. An external etcd can aid in maintaining control plane HA but won’t save your database setup. Keeping things straightforward might be your best bet.
Is there a way to deploy an external etcd with only 2 nodes?
It might be worthwhile to add a small third node if your budget allows. That could provide a necessary quorum for your control plane and improve your PostgreSQL leader election process.
Keep in mind that traditional PostgreSQL doesn't support true active-active configurations. You can set up an active-passive structure instead. With only 2 nodes, I suggest just running one instance and relying on your virtualization for failover, which usually won’t impact your operations significantly.
CNPG is useful, but be careful! With just two masters, you double the risk of failure; avoid having an even number of masters if possible.