My company runs an on-premises Kubernetes cluster with 64 GPUs. Our machine learning engineers need temporary development environments where they can write code, test it, and train models. We currently create Docker-in-Docker workloads dynamically, and engineers connect to the workload containers over SSH. Occasionally, they need access to the host kernel for GPU profiling or other low-level experiments. Giving them that access could let them change kernel or driver settings and affect unrelated workloads on the same node. Is there a safe way to provide this capability without compromising other users?
3 Answers
Put these workloads on a dedicated GPU node pool and taint those nodes so normal workloads cannot land there. Give the development workloads the matching toleration and node affinity. This does not make host-level access safe on a shared node, but it limits the blast radius. Treat those nodes as disposable so they can be drained and reprovisioned if someone damages the host.
It depends on what kind of access is actually required. If engineers only need GPU profiling permissions, you may be able to grant a narrowly scoped capability instead of exposing the host. However, GPU profiling settings can be node-global; for example, NVIDIA performance-counter access may depend on the NVreg_RestrictProfilingToAdminUsers driver setting. That is another reason to keep profiling workloads on isolated nodes rather than relying only on namespaces or containers.
If you genuinely need kernel isolation, regular containers cannot provide it because containers share the host kernel. Use a VM-based approach such as KubeVirt or Kata Containers, accepting the added operational complexity and possible GPU passthrough limitations. Tools such as virtual clusters can help isolate Kubernetes tenants, but they do not isolate the underlying operating system kernel, so they are not sufficient for untrusted kernel-level experiments.
Understood. I’ll investigate Kata Containers, while also considering a separate tainted GPU pool for workloads that need profiling.
That makes sense, but I was worried about a user doing something severe enough to break the whole node. Making the profiling nodes disposable may be the practical answer.