Hey folks! We're a research group that just got a really big NAS unit with 34 disks, each having 20TB of storage. We're aiming to centralize all our research data, which is currently spread across several smaller servers (about 2TB each), as well as store our service data using Longhorn deployed via Kubernetes. I haven't dealt with such a large setup before, and I'm unsure about which file system would be the best for us. I've done some digging but it seems like ext4 isn't the right choice for this capacity. We have a MegaRaid 9560-16i with 8GB for RAID, currently configured to have two RAID6 drives totaling 272TB, but I'm open to changing that up if needed. Here are the specs: CPU is an AMD EPYC 7662 (64 cores), and we've got 512GB of DDR4 RAM. Looking forward to your suggestions!
4 Answers
Sweet NAS setup! Just a friendly reminder to prioritize your backup strategy to avoid any data loss!
You might want to consider how you plan to back this all up, especially at such a large scale. Backups are crucial, and there's no one-size-fits-all solution, so start thinking about your strategy early!
ZFS could be a solid choice for your setup, especially given your hardware. It's great for handling large volumes of data, but keep in mind it works best with plenty of RAM and SSDs for a performance boost. Just make sure your HDDs are passed directly to the OS without RAID, through an HBA in IT mode, for optimal results.
Good to know! Just a heads up, ZFS really prefers direct disk access.
With the size of your NAS, exploring a SAN setup (like iSCSI or FibreChannel) might also be beneficial if you're looking for scalability and flexibility.
Thanks for pointing that out! I actually forgot to mention, we've got 2 1TB NVMe drives just for the OS.