I'm trying to find a practical way for workloads in a small, air-gapped Kubernetes cluster to access existing NFS data while still honoring POSIX permissions between multiple tenants. The data is too large to move into cluster storage, is actively used outside Kubernetes, and cannot be modified with chmod or chown.
The cluster runs Ubuntu 22.04 with RKE2 and Rancher on three control-plane and three worker nodes. It is a single-cluster, multi-tenant environment, with all nodes on the same subnet. Kyverno, NeuVector, Argo CD, and Longhorn are available, although Longhorn is mainly useful for new deployments. The NFS servers run AlmaLinux 8, use NFSv4 with sec=sys and manage-gids, and obtain UID/GID information from 389 Directory Server. The data is spread across several NFS servers.
The intended approach is to use manually defined NFS PersistentVolumes and PersistentVolumeClaims for existing directories, with tenant-specific policies and RBAC preventing tenants from creating or modifying PVs. I also have a ConfigMap containing tenant UID, GID, and group information.
Changing pod UIDs or GIDs is risky because some applications expect particular identities, and supplemental groups do not appear reliable with manage-gids. Re-exporting mounts through another NFS server and using fsid or squash options also seems fragile, especially concerning locking and kernel support. I would prefer not to change the existing production NFS exports, but I could create tenant-specific exports with all-squash if that is genuinely the only safe option.
Is there a better pattern for providing multi-tenant, POSIX-aware access to existing NFS data from Kubernetes, or are tenant-specific identities and exports unavoidable?
3 Answers
Object storage such as MinIO or RustFS avoids the POSIX identity problem entirely, since applications authenticate to buckets rather than mounting shared filesystems. That can work well for documents and other flat objects, but only if the applications support an S3-compatible API. A userspace filesystem mount is possible in some cases, but it introduces another layer and may not provide the performance or locking behavior your workloads need.
For databases or workloads that truly require filesystem semantics, block storage or a properly designed clustered filesystem is generally a better fit than trying to make NFS act as a Kubernetes-native tenant boundary.
The fundamental limitation is that NFS with sec=sys trusts the UID and GID values presented by the client. Kubernetes pods can present different identities, but the NFS server does not provide a clean tenant boundary based on the namespace or workload. Without changing the export configuration or using a storage system that enforces identities independently, ownership can effectively bleed between tenants.
Kerberos can provide stronger authentication and identity mapping, and it does not require Active Directory—FreeIPA is one possible management layer. However, Kerberos solves authentication and identity consistency more than it solves per-tenant storage isolation, so it would still need to be designed carefully for this use case.
If you can change the storage backend, a system with storage-side identity enforcement is a cleaner fit. For example, AWS EFS access points can assign a fixed POSIX UID/GID and directory root per access point, ignoring the identity claimed by the client. That gives each dynamically provisioned volume a separate filesystem-level identity and path. This only applies if the cluster can use EFS, so it is not suitable for a completely disconnected bare-metal environment.
For an on-premises alternative, CephFS managed through a suitable CSI integration may provide more flexible multi-tenant controls than re-exporting NFS. It would still require deploying and operating a new storage platform, though.
Object storage is not automatically a performance improvement either. Systems like MinIO can introduce their own operational and throughput problems, so the application access pattern and workload requirements need to be tested before treating it as a drop-in replacement.