I've been asked to design a highly available, fault-tolerant Kubernetes environment for on-premises GPU workloads. The system may serve roughly 1,000 registered users, with around 50 active users per day, primarily running Jupyter notebooks with GPU time-slicing enabled. The main goal is production-grade reliability with as little day-to-day operational work as possible.
Our current setup uses MicroK8s, Longhorn, Calico, and Charmed Kubeflow, but it has recurring problems: Calico controllers restart, pods unexpectedly remain pending, the dqlite database sometimes reports "500 database is locked" during leader election, and tokens occasionally need manual cleanup. The hardware is also fairly old, although replacement servers are being considered.
Which Kubernetes distribution and architecture would be the best fit for this environment? I'm particularly interested in HA control-plane and worker-node design, GPU scheduling and time-slicing, storage, networking, and alternatives to MicroK8s with Longhorn and Charmed Kubeflow. I'm also a single person supporting the environment, so operational simplicity and reliable vendor or community support are important.
5 Answers
If the budget allows it, Red Hat OpenShift with OpenShift AI is worth evaluating. It provides a supported platform for GPU workloads and includes tooling for notebook environments, which could reduce the amount of integration and troubleshooting required compared with assembling Kubernetes, storage, networking, and Kubeflow components independently. It will cost more, but that may be justified when there is only one person available to operate the system.
There isn’t really a zero-operations option for an on-premises GPU platform serving this many users. Plan for multiple control-plane nodes, redundant GPU workers, reliable shared or replicated storage, separate management and workload capacity, monitoring, backups, and tested node replacement procedures. Cluster API with image-built nodes can provide a replaceable ‘cattle’ model, but it still requires solid Kubernetes and infrastructure expertise.
Talos Linux with Kubernetes could be a good fit if you want an immutable, tightly controlled operating system. Pair it with Cilium for networking and evaluate Longhorn v2 or OpenEBS Mayastor for storage. Test the exact versions together before committing, since storage compatibility can vary significantly between releases.
RKE2 with Cilium is another reasonable production option, especially if you want a more conventional Linux-based distribution and enterprise support options. It may simplify some areas compared with MicroK8s, but it won’t automatically fix instability caused by faulty hardware, storage, networking, or configuration.
Before replacing the entire platform, identify where the current failures originate. The dqlite database-lock errors point to control-plane or disk problems, while repeated Calico restarts could be caused by node health, networking, or resource pressure. A different Kubernetes distribution may expose the same underlying issues if the old servers or storage remain unreliable.
That makes sense. The hardware is old and replacement servers are being arranged, but I’m still comparing platforms so we can make a better choice when the new infrastructure is ready.