What are the best on-premises block-storage options for Kubernetes workloads that cannot use NFS?
Asked by
MellowCedar42
I'm running a self-managed Kubernetes cluster in an air-gapped private datacenter with 10 servers, each having roughly 100 GB of SSD capacity. At the moment, NFS is the only available StorageClass, but its I/O behavior is causing problems for Camunda 8/Zeebe and Elasticsearch. Both use storage patterns that are a poor fit for NFS, including RocksDB operations, fsync, atomic renames, and file locking, and I'm seeing the kinds of failures their documentation warns about.
The worker disks are nearly full, so Longhorn and local-path are not practical. Elasticsearch already needs more than 100 GB and continues to grow, so it cannot simply live on one node's local disk. I need centrally managed storage with SSD-like performance that still allows PVCs to move between workers rather than being permanently tied to a specific node.
Is iSCSI or another SAN-style solution the usual answer? Which CSI drivers and architecture would be reasonable, and how difficult are they to operate? Would NVMe over TCP, Ceph, or another option be better?
Answers
If you cannot add shared block storage immediately, a temporary compromise would be to reserve a small amount of local SSD for Zeebe, since its active data footprint may be smaller than Elasticsearch’s. Elasticsearch could remain on NFS only as a short-term measure with conservative expectations, but that is not a good long-term design for a growing data set.
A distributed filesystem such as SeaweedFS is worth considering only if the applications explicitly support its semantics. It should not be assumed to be a drop-in replacement for block storage: Elasticsearch and RocksDB-based workloads need storage behavior that has been tested and documented for those specific applications.
Reply to this answer
iSCSI is probably the most practical fit here. A storage server or SAN exports LUNs, each worker connects as an iSCSI initiator, and a Kubernetes iSCSI CSI driver provisions and attaches block volumes to whichever node runs the pod. That gives the applications normal block-device semantics instead of NFS’s network filesystem behavior.
The setup is not trivial: you need reliable target and initiator configuration, multipath and failover planning, authentication, monitoring, backups, and careful capacity management. Also verify that the storage platform and CSI driver support the access modes your workloads need. If possible, dedicate SSD-backed storage to Zeebe and Elasticsearch rather than putting them on a general-purpose NFS share.
Reply to this answer
NVMe over TCP is another option if you have suitable storage hardware or software. Conceptually it is similar to iSCSI—centralized network-attached block devices with initiators and targets—but it can provide lower overhead and good performance over ordinary Ethernet. It still requires a properly designed, redundant storage target and a CSI integration that you are comfortable supporting.
Ceph or a similar distributed storage platform can also provide block volumes, but it generally needs dedicated disks, sufficient capacity, and several failure domains. With nearly full worker disks, deploying Ceph on those same nodes is likely to create more operational and capacity problems rather than solve them.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions