Best Strategy for Setting Up an On-Prem Kubernetes Cluster

Welcome to the Kubernetes adventure! You might want to try K3S if you're looking to save resources; it comes prepackaged with a lot but is still flexible for adding elements you need later. Just keep in mind that switching from a single control plane to a high-availability setup can become quite a task. If you're aiming for WebSocket operations, most setups should work fine, but investing time in Gateway API could be a smart move for future-proofing your cluster. Happy building!

From what I’ve seen, you’d be fine with one control plane and two workers. Lightweight versions like K3s can reduce many future headaches. I do have a couple of questions though: how production-ready are you aiming to get? Are you looking to learn or run a homelab? Consider using KVM to add some virtualization; it allows for HA configurations and creates more learning opportunities down the line. For the networking components, think about using Cilium and Gateway API with K3s.

Hey! Having multiple control plane nodes is great for managing upgrades and changes without downtime, especially in the early phases of your setup.

Using three control plane nodes is ideal for high availability, but if you don’t need constant access for kubectl or API services, a single control plane should suffice. For simplicity, I recommend trying out k0s; it's straightforward with just one binary to manage and works fine with its default CNI!

We're currently just planning for production with WebSocket deployment, and we might add Kafka later if needed. We can consider adding more nodes in the future!