After facing some unfortunate disasters in our company, we're in the process of rebuilding our OKD clusters. This situation gives us a chance to make some much-needed improvements, and we're definitely looking to move away from NFS due to numerous performance issues we've encountered. While we do have VSphere, our finance department has been unwilling to allocate funds for expensive options like VMware vSAN.
We've dabbled a bit with Ceph (using Rook) and set up a proof of concept on three VMs, but it seems quite complex to manage, and it appears to really require solid hardware to operate effectively. Longhorn's architecture also seems to rely on NFS when RWX is enabled, which raises some red flags based on mixed reviews we've seen regarding responsiveness.
We're searching for other storage solutions that are straightforward to maintain without breaking the bank. If you have any recommendations, please share your experiences, not just simple suggestions—context is appreciated!
8 Answers
In my experience, sometimes simple solutions work best. I've been using host path mounting directly for my containers. It's straightforward and saves a lot of headaches with complicated setups. Just ensure your pods mount the same file location across hosts.
I totally get your struggles! That's actually why we created Simplyblock—to tackle issues like unstable NFS performance and the complexity of solutions like Ceph. Simplyblock is designed to work without special hardware and aims for high performance and ease of use. It's definitely worth checking out if you want a cloud-native option!
Besides Ceph, have you looked into Portworx? They have a community operator available and a free tier that could fit your needs. It's worth considering!
Ceph using Rook is one way to go, but I've found it pretty complex. Rook simplifies some things, but you'll still need time and a solid understanding to operate it effectively. It can be daunting even after a proof of concept, so make sure to consider that if you're looking for something straightforward.
Finding a good distributed storage solution is tricky, especially if you're trying to keep costs down. Unfortunately, there isn't a magic bullet out there. You can either invest heavily in a SAN solution and let them handle support, or you might look into using a blob store for file storage, though that can come with its own challenges.
If you're using VSphere, I recommend giving VXFS a shot. It works seamlessly with OpenShift and should be compatible with OKD too. Just a heads up, you'll still need to figure out a RWX solution, but VXFS typically does well in that environment.
Sometimes it's better to go back to basics. If rebuilding is tough, maybe stick with NFS for now and migrate later. Other solutions like OpenEBS and DRBD might be worth exploring, but don't put too much stress on the rebuild if NFS can hold you for now.
Just throwing this out, but if you're worried about stability and redundancy, you might want to consider Piraeus/LINSTOR for replicated storage. It's user-friendly and performs well if you don't need RWX right off the bat.
Does it support RWX?