We have an on-prem Kubernetes cluster with 64 GPUs. Machine learning engineers use dynamically provisioned Docker-in-Docker pods for development and training, connecting to the workload containers over SSH.
Some engineers also need kernel access for GPU profiling and experiments. How can we give them what they need without letting changes affect other workloads on the same node?
Start by separating profiling permissions from actual kernel isolation. Do the engineers need specific capabilities or access to profiling interfaces, or do they need to change kernel parameters independently? Those are different requirements.
Containers on a node share the host kernel. A virtual cluster may help organize tenant resources, but it does not give each team a separate kernel. If different teams need to make conflicting kernel changes, look at VM-based isolation, including options such as Kata Containers or KubeVirt.
A dedicated node for these experiments is the straightforward option. Taint it to keep ordinary workloads off, then give the development workloads the matching toleration and node affinity.
That protects unrelated workloads from node-level changes, but it doesn't stop an experiment from breaking the dedicated node itself. Treat it as a machine that may need to be restored, not as a safely shared node.
If you need VMs, consider running them on a regular hypervisor rather than automatically putting them inside Kubernetes. KubeVirt can provide that integration, but it also adds another layer to operate and debug. Whether that tradeoff makes sense depends on what your team is prepared to manage.
Answers are reviewed before publication.
cabbage referee:
Good distinction. We need to check which profiling tasks actually require host-level access before choosing an isolation model.