I'm currently in the middle of a debate about our infrastructure as we transition from Hyper-V to Proxmox. We have a straightforward setup with 8 nodes (4 primary and 4 backup). Historically, we've used dedicated storage directly in our machines, but someone is suggesting that we switch to shared storage via a SAN. I'm not sure that's the best route. Right now, each server has 8x 4TB NVMe drives that mirror to each other, and those servers replicate to their backups every 10 minutes. This means if the primary server fails, we can quickly boot up VMs on the backup without a hitch.
With shared storage, everything would depend on the SAN, which sounds like a single point of failure to me. How is shared storage better in this case? Am I missing something here?
6 Answers
You're totally spot on. Putting everything on a SAN really does feel like putting all your eggs in one basket. A SAN may look sophisticated, but if it fails, your whole system goes down. Plus, with dedicated storage, you’re getting way better IOPS. Your current setup is solid with redundancy and quick recovery. I'd say stick with what you have! No need to chase after what’s shiny when your system works well.
It's true that SANs usually come with redundancy like dual controllers and power supplies, making them pretty resilient. But, as you said, that's not foolproof. The risk of a single point of failure is still a concern, especially for smaller setups.
If you do go for any clustering, don’t forget to add a quorum device! Even-numbered clusters can create issues down the line, so it’s something to think about.
Another angle to consider is storage size. When you need more storage than what a single server can handle, a SAN might become necessary. But if that’s not the case for you, then internal storage seems like the better route here.
In the past, SANs had their benefits, especially with expensive solid-state storage where costs could be shared. But with today's tech, internal storage wins hands down. You avoid SPOF, get better performance, and there’s no vendor lock-in. Running a small cluster like yours, I'd definitely advise against using shared storage.
Have you considered using Ceph instead of a traditional SAN? It offers the benefits of shared storage without the costs associated with dedicated SANs. With 8 nodes, you have plenty of redundancy, and it'll utilize all your SSDs for performance boosts. Definitely worth looking into!
Thanks for the tip! I'll definitely check out Ceph.