I've worked with ODF (Red Hat's paid version of rook-ceph, or something similar) and Portworx in larger enterprises, but now I'm searching for open-source or low-cost options for on-cluster replicated storage. ODF and Portworx don't fit the budget, so I'm turning to the community.
My setup includes a 3-node baremetal cluster that can handle scheduling for both worker and control plane tasks. The storage layout consists of a RAID1 boot pool on SSDs and either a RAID6 SSD or HDD pool for actual data storage.
I've tested a few solutions and have some hesitations about bringing them into production:
- **Longhorn** v1 and v2 show promising performance, especially v2, but its general stability is concerning. I've noticed that a node crash or even a simple reboot during a k8s upgrade can lead to volume loss.
- **Rook-ceph** is pretty resilient but comes with complexity that makes it hard to fully grasp, plus random read performance wasn't impressive during benchmarks.
- **OpenEBS** performed well in benchmarks and failed recovery, but it took too long to initialize large block devices (like 10 TB) and lacks native RWX volume support.
- **CubeFS** didn't perform well in benchmarks, possibly because it isn't built for smaller clusters like mine.
3 Answers
I’ve been reflecting on storage options lately, and I’ve concluded that replicated storage may not be as essential for most applications, except maybe for virtualization setups like kubevirt. Most modern apps rely on databases, S3, or brokers, and those manage their replication on their own. Since we want to limit the risk of a drive failure, I'm focusing on how to best pool and assign the drives to maintain performance while still having some redundancy.
Not a free or open-source option, but I've seen Weka and LightBits recommended for storage solutions. They might be worth looking into if you have the budget.
I’ve been using Longhorn on a Talos cluster with good success. I upgraded both Talos and Kubernetes recently, and things went a bit off track during the process. There was a note in Longhorn's docs regarding a mobile flag during the upgrade that I overlooked, which caused some panic. I ended up downgrading and re-upgrading to a version that included the iscsi plugin, and everything was back to normal after that. Just a heads up if you go this route!
Unfortunately, our applications can't accommodate other storage systems and we can't rely on external solutions at the edge. I think replicated storage might be our best bet to allow for pod mobility and to protect against hardware failures.