I run a production Kubernetes environment without access to cloud storage and currently use nfs-utils with the nfs-server service for shared storage. Over time, I've experienced several serious problems: worker nodes have developed dead or queued NFS processes, causing very high system load despite low CPU and memory usage; df -h has hung; and NFSv4 sessions have appeared to leave requests blocked when the server or connection has problems. In one incident, nfsd reported blocked threads, restarting the service also hung, and the worker had to be rebooted. Are these common NFS failure modes in Kubernetes? What diagnostics, configuration changes, or on-premises alternatives—such as Longhorn, Ceph, or a SAN with a CSI driver—would make this setup more reliable?
4 Answers
For Kubernetes workloads that need shared state, consider whether every workload really requires a shared filesystem. A replicated block-storage system such as Longhorn or Ceph may be a better fit for many applications. For cases such as one process renewing certificates while another reads them, shared storage can still be valid, but you should also consider a dedicated secret-management or certificate-delivery mechanism with clear update and reload behavior.
The symptoms can be caused by kernel or NFS client/server compatibility problems, so check that the worker and NFS server kernels, nfs-utils packages, and mount options are current and consistent. A hanging df usually means a process is stuck in uninterruptible sleep while waiting for an NFS response. That can cause requests and processes to accumulate until the node becomes unhealthy.
NFS server thread limits and client mount behavior are worth reviewing. The default nfsd thread count may be too low for many pods, so measure the workload and increase it carefully—for example, testing a value such as 32 instead of the default 8. Be cautious with changing hard mounts to soft mounts: soft mounts can return errors instead of blocking indefinitely, but they may also cause failed or corrupted writes. If you use them, set conservative timeo and retrans values and only for workloads that can safely handle I/O errors. Fixing the underlying network, server, and kernel issue is preferable to masking it.
NFS often isn’t the best first choice for production Kubernetes storage. If you have access to a SAN, its CSI driver is usually a stronger option. For a smaller self-hosted environment, Longhorn can provide replicated volumes and a simpler operational experience. Ceph is another capable choice, especially when you need scalability and are prepared to invest more time in deployment and maintenance; Rook can help manage it through Kubernetes.
That’s the kind of case I had in mind: one job writes renewed certificates while a load balancer reads and reloads them. I’ll look at whether a certificate or secret-delivery solution can avoid keeping both jobs on the same NFS mount.