I'm part of a small startup with a tight budget, and we need a reliable on-premise server for various AI tasks like LLM inference, TTS, and STT. Since going for a high-end Threadripper or EPYC setup isn't feasible for us, we're planning to build a solid workstation that can function as a server. Our aim is to create a machine that runs two RTX 4090 GPUs for inference requests, capable of operating 24/7 without maxing out at 100% load all the time. Here's a breakdown of the components we're considering:
- **CPU:** AMD Ryzen 9 9950X
- **Motherboard:** ASUS ProArt X670E-Creator WiFi (supports x8/x8 PCIe Bifurcation)
- **GPUs:** 2x RTX 4090 (likely either MSI Gaming Trio or ASUS TUF/ROG depending on availability)
- **RAM:** 96GB Corsair Vengeance DDR5 6400MHz (planning to downclock)
- **Storage:** Samsung 990 PRO 2TB NVMe
- **PSU:** Corsair AX1600i Titanium (1600W)
- **Case:** Corsair 7000D Airflow (White)
- **CPU Cooler:** Arctic Liquid Freezer III 420mm
- **UPS:** 3000VA APC/Eaton UPS
We know fitting dual thick GPUs on a standard ATX motherboard is tricky. To avoid thermal throttling on the top card, our plan is:
1. Top GPU mounted horizontally in the first PCIe slot.
2. Bottom GPU mounted vertically using the Corsair 7000D's vertical slots and a quality PCIe 4.0 Riser Cable.
3. Ensure plenty of ventilation.
I have a couple of specific questions:
1. Is the combination of one horizontal and one vertical GPU suitable for a server environment? I'm a bit concerned about the potential instability from the riser cable.
2. We're considering limiting the power to about 70% for both cards (around 300-315W) to help with thermal management and longevity. Is this approach advisable for production inference workloads? Any tips or red flags before we finalize our build would be really helpful!
1 Answer
Have you already purchased those RTX 4090s? It's worth mentioning that they're hard to come by now and prices have skyrocketed. You might want to consider going with a single RTX 5090 instead since it could be easier to handle and more cost-effective in the long run. Plus, that card offers some advantages as well. Just a thought!
I appreciate the suggestion! However, the 5090 is about $700 more than a 4090 and has only 32GB of VRAM compared to 2x 24GB with the dual setup. I'm really more focused on making sure this build is reliable and can actually handle the workload. Thanks for your insight!