Hey everyone! My team manages several Kubernetes clusters for different groups within our company, and we have set strict policies to ensure specific permissions. We generally limit actions like running root containers, creating Custom Resource Definitions (CRDs), and installing DaemonSets to maintain security. However, some teams are now asking for more elevated permissions to deploy their applications, which includes creating ClusterRoles and using CRDs. I'm looking for advice on how to grant these permissions without compromising the security of our clusters. Has anyone tackled this before? Any best practices or insights would be hugely appreciated!
4 Answers
It really depends on how your current clusters are set up. One option is to use virtual clusters via vCluster, giving each team their own API server for conflict-free CRD management. Another approach is a graduated permissions model where teams could request elevated permissions using things like admission webhooks with OPA Gatekeeper or Kyverno, maybe even with time-limited elevation policies. It could help keep everything secure while still being flexible enough for team needs. Just ensure you have proper network policies and resource quotas in place!
There's no one-size-fits-all solution here. I think it's crucial for the platform team to collaborate with the teams making these requests to really understand the need behind them. Often, the requested permissions may not actually be necessary if there's a way to run things in a namespaced manner. Testing could also be done in isolated environments where teams could control things, and then come back with a stronger case for what they need.
Not many workloads actually need to run as root. Often, it’s just a developer's convenience that gets mistaken for a requirement. For cluster-wide permissions, it makes sense to limit those to dedicated clusters as opposed to shared ones, where it's a definite no-go.
A solid practice is to let cluster admins manage CRDs through GitOps, where tenants can submit PRs for changes. This centralizes control and reduces conflicts. For roles and bindings, consider a self-service layer if you haven't put one in place yet. As for root containers, I'd recommend not allowing them at all, but if it's a must, at least isolate that risk by using tenant-specific node pools or clusters. You could also allow specific kernel permissions while preventing general root access.
Capsule can also be a nice alternative to the Hierarchical Namespace Controller for managing all that.