How to Achieve Hardware Isolation for ML Models in Kubernetes?

You might not need to abandon Kubernetes. Instead, consider allocating dedicated bare metal hardware for each tenant's cluster. It could be more secure, but yes, it can also get pricey.

Have you considered creating dedicated node pools with tainted nodes? It’s not a perfect solution, but it provides better isolation than a multi-tenant setup.

If hardware isolation is a revenue-generating requirement for you, I’d also explore some paid solutions. vCluster might not fit your exact needs, but their backers offer multi-tenant specific solutions which could be interesting. There's also cutting-edge tech like what VNode provides, focused on true hardware-level separation.

The issue of Kubernetes isolation is becoming more significant due to stricter security requirements. You’re spot on—namespaces offer limited protection. Have you looked into confidential containers that use hardware for isolating workloads even from the host OS? It takes a bit of setup and special node configurations, but it’s feasible. We eventually transitioned our sensitive workloads outside of Kubernetes to achieve true hardware isolation, using confidential VMs that Kubernetes interacts with through APIs. It’s more complex but much safer.

Simply labeling nodes and using selectors might not meet your needs. Binding clients to specific hardware often complicates cloud scaling, but it’s worth exploring if you want to hasten your data center expansion!

One way to tackle this is by using a mutating admission controller. You could enforce the presence of a `nodeSelector` for any pods within isolated namespaces. If you've set up logical isolation already, employing nodeSelectors that correspond to those namespaces and labeling the nodes accordingly could help. With tools like cluster autoscaler, you can even dynamically manage nodes for each tenant's namespace.