I'm looking to improve how we manage our cluster setups. Right now, we have a 3-node cluster with one control plane and two worker nodes, using namespaces for each environment. However, I want to transition to having multiple clusters for each environment and reserve namespaces just for deployment workloads. We have some bare-metal servers available within the same network and I'm interested in finding a platform that allows us to configure and provision new clusters on-the-fly. I've heard of Rancher and think it might be a solution, but I'd love some advice on this. Any suggestions?
5 Answers
If you're dealing with Kubernetes, you might want to check out virtual clusters. They could fit what you're trying to achieve. Alternatively, Kamaji is a more complex solution that facilitates running cluster control planes as pods and allows you to easily add nodes. It might suit your needs better, so I'd suggest giving that a look!
I live by Cozystack in my home lab! It uses bare metal to create a single cluster and adds nodes as you get more. Each ‘tenant’ can run their own Kubernetes clusters with autoscaling VMs, which sounds a lot like what you're describing. Plus, it can create new clusters on-the-fly using Cluster API and manage workloads efficiently during downtime. It's a bit complex but runs everything smoothly for me. Check it out at cozystack.io!
What you’re after sounds a lot like Omni, which is designed specifically for dynamic cluster management. Check out siderolabs.com/omni and let me know if you have any questions about it!
Using Rancher with VMware is super straightforward! Adding new control or worker nodes is a breeze, plus altering their sizes is quite simple too. I've only used it in a virtual setup, but previously, when I had to add clusters on-the-fly, it was pretty easy to manage. Just a heads up, rolling out new VM templates can be slow unless you're on RKE2, where automation helps a lot!
You might not find a ready-made solution for this, but you can write something using the Cluster API. That said, you'll need to provision a control plane node first before you can really make use of the API to manage clusters. If you're looking into doing things with libvirt/kvm, that might work for provisioning new VMs as Kubernetes nodes, but it sounds like you're searching for a platform solution.
Could you explain more about leveraging the Cluster API? If I prefer to set up a control-plane, then I believe Cluster API isn’t needed at the start. Just curious about how that might fit into your plan.