I've been asked to design a highly available, fault-tolerant on-premises Kubernetes environment for GPU-based workloads. The target is roughly 1,000 registered users, with about 50 users per day, mostly running Jupyter notebooks. GPU time-slicing is enabled, and the goal is to keep daily operational work as low as possible without sacrificing production reliability.
Our current setup uses MicroK8s, Longhorn, and Charmed Kubeflow, but it has recurring problems: Calico controllers restart, pods unexpectedly remain Pending, and the control plane sometimes struggles with dqlite database-lock errors during leader election. Tokens occasionally fail to refresh, requiring manual cleanup. The hardware is also relatively old, although replacement servers are being considered.
What Kubernetes distribution and architecture would be a good fit for this environment? I'm especially interested in practical recommendations for highly available GPU nodes, networking, storage, and a Kubeflow or notebook platform with reasonable operational overhead. Are there good alternatives to the current MicroK8s, Longhorn, and Charmed Kubeflow combination?
5 Answers
Before changing distributions, isolate the current failure modes. The dqlite database-lock errors and repeated Calico restarts point to control-plane, storage, hardware, or resource problems rather than necessarily a MicroK8s limitation. Check disk latency and health, memory pressure, CPU saturation, network reliability, time synchronization, control-plane quorum, and the events for Pending pods. A different Kubernetes distribution may reproduce the same issues if the underlying hardware or configuration is unstable.
RKE2 with Cilium is another practical distribution to evaluate, but it won’t automatically solve the existing failures. Its value is the relatively conventional, supportable production setup. Build a small test cluster first, validate GPU scheduling and time-slicing, notebook lifecycle behavior, storage recovery, node replacement, and control-plane failure scenarios before migrating the full environment.
OpenShift with its AI and notebook components is a strong supported alternative if the budget allows it. It provides an integrated platform and commercial support for many of the pieces that otherwise have to be maintained separately. The trade-off is licensing cost and adopting the platform’s conventions, so confirm that it fits the customer’s requirements before choosing it.
Talos Linux with Kubernetes is worth evaluating, especially if you want immutable nodes and a more consistent operational model. Pair it with Cilium for networking and compare Longhorn v2 with OpenEBS Mayastor for storage. Test the exact versions you plan to deploy—storage compatibility can change between releases, and older Longhorn versions have had issues.
I’m familiar with Talos, so I’ll investigate that combination and test both storage options against the actual GPU and notebook workloads.
There isn’t really a low-operations option for a self-managed, highly available GPU platform. You’ll need reliable hardware, redundant control-plane nodes, separate or carefully designed storage, tested backups, monitoring, upgrade procedures, and someone responsible for incident response. If the team is small, a supported platform or a managed service may be more realistic than assembling every component yourself.
The workload is mostly Jupyter notebooks rather than high-volume inference, but I agree that hardware quality and operational ownership are still important even at that usage level.
That makes sense. The control plane appears to get stuck during leader election, and the old servers may be contributing. Replacement hardware is being discussed while I continue troubleshooting.