I'm trying to give workloads in a small, air-gapped Kubernetes cluster access to large existing NFS datasets while preserving the current POSIX ownership and permissions for multiple tenants. The cluster runs Ubuntu 22.04 with RKE2 and Rancher, has three control-plane servers and three workers on the same subnet, and is intended to serve several vendors from one cluster. Kyverno, NeuVector, Argo CD, and Longhorn are available, although Longhorn cannot hold the existing data.
The NFS servers run AlmaLinux 8 and use NFSv4 with sec=sys, manage-gids, and identity data from 389 Directory Server. The data is spread across several servers and is heavily used outside Kubernetes, so changing ownership or permissions is not acceptable. I would prefer to use static in-tree NFS PVs and PVCs for existing directories rather than deploy a dynamic provisioner.
The main constraints are that I cannot modify the live NFS data, pods must not be able to choose arbitrary UIDs or GIDs, and supplemental groups have not worked reliably with manage-gids. I have considered mutating pod security contexts, creating tenant-specific exports with all-squash, or re-exporting mounts through a separate NFS server, but each option is either fragile, highly manual, or potentially problematic for locking and kernel support.
What is the cleanest way to provide tenant-isolated, POSIX-respecting access to these existing NFS volumes from Kubernetes without changing the data or relying on every workload using a particular UID and GID?
3 Answers
Kerberos does not require Active Directory. FreeIPA can provide the KDC, LDAP identity management, and host integration, although deploying and operating it in an air-gapped environment adds complexity. It can make user and group identity more consistent across clients, but it still does not solve the problem of separating tenants that share the same NFS namespace. You would still need export, directory, or storage-level isolation.
The fundamental limitation is that NFS with sec=sys trusts the UID and GID presented by the client. Kubernetes pods can present different identities, so NFS by itself does not provide a strong tenant boundary in a shared cluster. Kerberos can authenticate clients and improve identity handling, but it does not automatically create per-tenant filesystem isolation.
The strongest pattern is a storage-layer identity override, such as EFS Access Points. Each access point can enforce a configured POSIX UID/GID and directory, regardless of what the client requests. The CSI driver can provision one access point per volume and pair it with tenant-specific pod security settings. That avoids changing existing file ownership. However, it is only applicable if the environment can use EFS or a comparable platform service.
For on-premises infrastructure, CephFS with carefully designed client isolation is a closer equivalent, though it is a substantial deployment. If the applications can be changed to use object storage, MinIO or RustFS avoids POSIX identity problems entirely, but mounting object storage through a userspace filesystem may introduce its own performance and consistency issues.
There may not be a magic Kubernetes setting here. With existing NFS data and strict POSIX semantics, the practical choices are to enforce a known identity at the export or storage layer, or accept that workloads can claim identities. Tenant-specific exports with root or all-squash can work when the number of tenants is small, but they need to be managed carefully and generally limit each export to one tenant or one shared group.
Object storage is often a better fit for documents and other flat files because applications authenticate to the service instead of inheriting host filesystem permissions. It is not a drop-in replacement for databases or applications that require real POSIX semantics; those workloads usually need block storage or a filesystem designed with tenant isolation in mind.
That matches my concern: replacing the existing storage would require application changes, while the current workloads expect normal filesystem behavior. The export-based approach may be the most realistic short-term option despite the operational overhead.