I'm running a self-managed Kubernetes cluster in an air-gapped private datacenter with 10 servers, each having roughly 100 GB of SSD capacity. At the moment, NFS is the only available StorageClass. I'm deploying Camunda 8 with Zeebe and an Elasticsearch/ELK stack, but putting their PVCs on NFS is causing I/O problems and the filesystem behavior is triggering the issues these applications warn about. Zeebe relies on RocksDB and needs reliable fsync, file locking, atomic rename, and POSIX semantics, while Elasticsearch also performs poorly on NFS. The worker disks are nearly full, so Longhorn and local-path are not currently practical, and the Elasticsearch data already exceeds 100 GB and continues to grow. What centralized, SSD-backed storage options would let PVCs move between workers without being tied to a particular node? Is iSCSI or another SAN-style solution the normal approach, and which CSI options are worth considering?
4 Answers
A distributed storage system such as Ceph can provide Kubernetes block volumes, but it normally needs several dedicated disks or nodes for OSDs, plus memory and network headroom. With only about 100 GB per server and disks already close to full, adding Ceph may make the cluster less reliable rather than solving the problem. SeaweedFS or similar systems are only suitable if the application supports their storage interface; they are not a drop-in replacement for POSIX block storage used by Elasticsearch or RocksDB.
The key distinction is that you need shared or network-attached block storage, not simply another shared filesystem. A SAN or iSCSI array with a compatible CSI driver is the conventional route. Make sure the storage system supports snapshots, reservations or proper fencing, multipath connectivity, and reliable failover. Also confirm the Elasticsearch and Zeebe support requirements for the selected volume mode and filesystem instead of assuming any block device will be safe.
NVMe over TCP is another centralized block-storage option. Conceptually it is similar to iSCSI, using a target and initiators, but it can provide a cleaner and faster path when the storage hardware and network support it. Kubernetes integration is less standardized than iSCSI, so verify that the CSI driver is actively maintained and test node failure, volume attachment, multipathing, and fencing before relying on it in production. NVMe over RDMA can be excellent but is considerably more complicated than NVMe/TCP.
iSCSI is probably the most practical answer here. It presents block devices rather than a shared filesystem, so RocksDB and Elasticsearch get the storage semantics they expect while volumes remain reachable from different Kubernetes workers. You’ll need an iSCSI target on the storage system and an initiator configured on every worker, followed by an iSCSI CSI driver in Kubernetes. It adds some operational work—multipath, authentication, capacity planning, monitoring, and recovery testing—but it’s a common and workable setup. If you can eventually dedicate disks to it, Ceph or another distributed block-storage platform could also work, although your current free capacity sounds too limited for that.
A small local SSD volume for Zeebe could be a temporary stopgap if it fits, but moving the Elasticsearch data to NFS would still leave you with a risky and slow arrangement. I’d treat that as a short-term bridge, not the final design.