Hey everyone! I'm in the process of setting up a high-performance server for a small organization, primarily for internal use where the team will be doing data analysis with tools like RStudio. As a junior systems admin without much experience in creating partitioning strategies, I'm seeking your advice to refine my plan.
Here's what I have to work with: I have a total of 4 NVMe disks (27.9TB each), adding up to around 111.6TB of space, alongside a 1.7TB OS disk (with 512MB allocated for /boot/efi and the rest for the / partition). I don't have the luxury of a test server.
**Current Requirements and Considerations:**
- My first dataset is expected to take up around 3TB, and I anticipate needing more space as different projects come in.
- I'm planning to allocate some temporary scratch space for processing.
- The partition setup should allow users to seamlessly work on their code while analyses are running.
- To keep it simple, I'm avoiding LVM and RAID for now, though I'm learning ZFS; my filesystems will mostly be ext4 and XFS since I'm more comfortable with those.
Here's my proposed layout:
- **Disk 1:** /mnt/dataset1 (10 TB, XFS) for initial datasets and future needs.
- **Disk 2:** /mnt/scratch (15 TB, XFS) for temporary files during processing.
- **Disk 3:** /home (10 TB, ext4) for RStudio user directories, and /results (10 TB, XFS) for analysis results.
- **Disk 4:** /backup (10 TB, ext4) for important files.
Additionally, I'm considering breaking down some of the OS partitions (like /tmp, /var, /var/log, and /var/log/audit) according to CIS recommendations, but I'm unsure about the appropriate space allocations to make.
What do you think of this setup? What are its strengths and what issues do you foresee?
4 Answers
I gotta agree with the previous comment. No redundancy is a huge concern! You should definitely start looking into backup solutions that aren't on the same hardware as your live data; it’s essential for any production setup. Also, be cautious about separating everything onto different disks; it can complicate things. A single volume or a proper LVM setup might simplify your workflow and allow for easier management as your data needs grow. Don’t forget to add some swap space too, just in case!
You need to prioritize redundancy in your setup. A single disk failure can halt everything! Also, keep in mind you lose a lot of flexibility without using LVM. You're going to want to resize partitions as your data grows, so consider integrating LVM sooner rather than later. With fast disks like NVMe, performance shouldn't be a big issue. Mirroring might seem to reduce space, but it's worth it for data safety. Lastly, ensure you have good backups in place before you start storing critical data.
I see a couple of potential red flags in your plan. First off, there's no redundancy, which means if one disk fails, you could lose data. Having all your backups on the same hardware is risky, even if they're on separate disks. It also looks a bit peculiar that you aren't using the full capacity of your disks from the get-go. Adding partitions later could lead to a mismatched setup. I would suggest considering a reliable backup system that's stored on separate hardware—maybe explore a cloud solution for backups. You may want to rethink your approach to disk space and look into using LVM for flexibility as your requirements grow.
I think you're on the right track with your partitioning needs, but I would suggest potentially simplifying a few things. Having everything on separate disks could lead to unnecessary complexities. Unless you have specific needs for distanced I/O, I'd recommend using larger volumes for ease of management. Utilizing LVM early can really save you a headache down the line when things start scaling up. And definitely consider planning out a solid backup solution!
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures