I'm planning a high-availability K3s cluster with three control plane nodes, two workers, a private API endpoint, and Cilium. I'd like to learn more about cluster operations as well as architecture.
I'm considering an immutable node OS, Gateway API for public services, and either a NAT gateway or public IPs with strict firewall rules for node egress. What would you change or pay particular attention to?
For egress, I'd keep the nodes private and use a NAT or egress gateway. A public IP on every node can be protected by firewall rules, but it also means an accidental rule change could expose a node. A dedicated outbound path is easier to audit.
Check the availability of that path too. A single NAT node can become a single point of failure even if the Kubernetes control plane is highly available. Keep public application traffic separate through a load balancer and gateway.
Talos Linux is another immutable option worth looking at alongside Flatcar and Kairos. It's focused on Kubernetes and managed through an API, so there may be fewer general-purpose OS pieces to deal with.
That said, a familiar Linux distribution is a reasonable choice if the OS is going to distract you from learning the cluster itself. You can always try an immutable OS in a separate build later.
Three control plane nodes with etcd is a sensible place to start, but don't stop at counting nodes. Plan around etcd quorum, backups, and how you'd recover a failed control plane. Test those procedures rather than assuming the extra nodes cover every failure.
Depending on the environment, combining control plane and worker roles on a small number of nodes can also be simpler. Separate workers are useful if you want to explore that operational boundary.
Cilium can do a lot more than provide pod networking. Network policies for both ingress and egress are worth exploring, and BGP is another option if your network supports it.
For Gateway API, check the features you need before settling on an implementation. Envoy Gateway is one alternative if Cilium's built-in gateway doesn't cover your requirements. None of that means you have to add BGP or every policy feature on day one.
Give the production cluster a non-production friend. Use it to test upgrades and configuration changes before applying them to the cluster that matters. High availability helps with failures, but it won't save you from rolling out a bad change everywhere.
Answers are reviewed before publication.
Related Discussions