I'm working with a mix of hardware for a virtualization environment that needs to handle about 30 medium-load VMs and around 40TB of data, split between the VMs themselves and other file data. My setup includes two Intel Silver servers with substantial RAM and storage, one Intel Bronze server, and a few SSDs and NVMe drives that are great for OS and cache but not suitable for an array. I've got 9460-16i RAID controllers but also have a spare 9300-8i HBA. My main concern is how to build a cluster while avoiding single points of failure, especially since I don't have a dedicated SAN. Here are a few ideas I'm considering:
1. **Distributed Storage:** I'm hesitant about options like GlusterFS due to wasted disk space on replicas and think Ceph might be overkill for my needs with fewer nodes.
2. **Simple Shared Storage Pool:** Setting up a NFS or iSCSI server on something like Rocky Linux, or using a system like TrueNAS or OpenMediaVault, but I'm worried about disaster recovery if that storage server fails.
3. **Local Approach:** Using local storage on my strong nodes with a backup strategy from the third node and possibly letting one of the local storages be shared for VMs.
I'd love to hear your thoughts or recommendations for my situation!
2 Answers
With your hardware, a distributed storage setup might not be ideal for a production environment. I'd recommend going with NFS, whether on vanilla Linux/BSD or TrueNAS. If you want a hyperconverged approach, you could also share local storage across all nodes. Just remember that whatever you choose, proper backup is key since replication isn’t the same as having a reliable backup solution. I'm leaning towards local storage on your two stronger nodes for ease of management and recovery.
And did I understand right that the 9460-16i is hardware RAID only? That complicates things a bit, right?
Before going deeper into setups, different setups come with specific recovery point objectives (RPO) and recovery time objectives (RTO), so you should clarify how critical those are for your applications. For your setup, I’d suggest using Proxmox across your three nodes with Proxmox Backup on the Bronze node. It’s a bit straightforward and has decent backup capabilities. While Ceph is powerful, three nodes are on the low end for it.
Yeah, oVirt is a must for us, but I'd have loved Proxmox for its user-friendly interface. It might complicate our existing plans though.
Have you thought about ZFS for backups? I know you can swap controllers for easier management, but that looks tricky.
What about disaster recovery though? Would it be easier to restore an entire Zpool from TrueNAS, or just bring back an NFS server on Rocky Linux?