I'm running a self-managed Kubernetes cluster in an air-gapped private datacenter with 10 servers, each having roughly 100 GB of SSD capacity. At the moment, NFS is the only available StorageClass. I'm deploying Camunda 8 with Zeebe and an Elasticsearch/ELK stack, but putting their PVCs on NFS is causing I/O problems and the filesystem behavior is triggering the issues these applications warn about. Zeebe relies on RocksDB and needs reliable fsync, file locking, atomic rename, and POSIX semantics, while Elasticsearch also performs poorly on NFS. The worker disks are nearly full, so Longhorn and local-path are not currently practical, and the Elasticsearch data already exceeds 100 GB and continues to grow. What centralized, SSD-backed storage options would let PVCs move between workers without being tied to a particular node? Is iSCSI or another SAN-style solution the normal approach, and which CSI options are worth considering?
2 Answers
NVMe over TCP is another centralized block-storage option. Conceptually it is similar to iSCSI, using a target and initiators, but it can provide a cleaner and faster path when the storage hardware and network support it. Kubernetes integration is less standardized than iSCSI, so verify that the CSI driver is actively maintained and test node failure, volume attachment, multipathing, and fencing before relying on it in production. NVMe over RDMA can be excellent but is considerably more complicated than NVMe/TCP.
iSCSI is probably the most practical answer here. It presents block devices rather than a shared filesystem, so RocksDB and Elasticsearch get the storage semantics they expect while volumes remain reachable from different Kubernetes workers. You’ll need an iSCSI target on the storage system and an initiator configured on every worker, followed by an iSCSI CSI driver in Kubernetes. It adds some operational work—multipath, authentication, capacity planning, monitoring, and recovery testing—but it’s a common and workable setup. If you can eventually dedicate disks to it, Ceph or another distributed block-storage platform could also work, although your current free capacity sounds too limited for that.
A small local SSD volume for Zeebe could be a temporary stopgap if it fits, but moving the Elasticsearch data to NFS would still leave you with a risky and slow arrangement. I’d treat that as a short-term bridge, not the final design.