Hey everyone! I'm gearing up to set up my first on-prem Kubernetes cluster with three Ubuntu 24.04 servers, and I'd love some help figuring out the best architecture. I'm weighing two options: having one control plane and two worker nodes, or setting all three up as a control plane with a stacked configuration. As I'm new to this, I'd appreciate advice on key components to install and configure, best practices for high availability, recommended network choices (CNI), and any common pitfalls to watch out for. I'm planning to deploy a Rust WebSocket application as well. Thanks for any insights or real-world experiences you can share!
5 Answers
I've found that having a few control plane nodes can be really helpful. It allows you to carry out migrations or changes without knocking everything offline, especially during the initial setup phase.
Most advice seems to lean toward a single control plane and two worker nodes, which should cover your needs. Also, considering how you plan to use this setup could influence decisions. Are you aiming for a production-ready environment, or is it more for learning? If it’s for production, adding a virtualization layer using KVM might allow you to better allocate resources and create a more resilient control plane setup. Just keep in mind this can complicate things a bit.
Welcome to the Kubernetes world! Have you considered K3S? It's lightweight and comes with a lot integrated, plus K3sup makes installation super easy. Just a heads-up, though; if you plan to scale to a highly available setup later, transitioning from a single control plane to HA can be a real headache—I had to start fresh with Velero for backups. Also, for WebSockets, any solution should work, but I'd recommend looking into the Gateway API for future-proofing your cluster. It's the direction Kubernetes is going!
Using three control plane nodes makes sense only if you need that high availability. If not, a single control plane should suffice. Since you’re using Ubuntu 24.04, have you considered k0s? It's incredibly easy to set up, just a single binary. Stick with the default CNI; it works well! My main tip is to focus on what you need and avoid getting bogged down with alternatives unless necessary.
Thanks for the suggestion! Sounds like K3S could really simplify things and I appreciate the heads-up on the challenges with HA.