I saw a listing for 300 used racks with 1,200 compute nodes, each advertised with eight A100 40GB GPUs. Beyond the roughly $59 million asking price, what would someone need to consider before using hardware like this for on-prem AI?
What would it take to put a used A100 rack listing to work?
Answers
Start with the facility, not the GPUs. That is 9,600 GPUs if the advertised quantities are correct, plus CPUs, networking, management nodes and power supplies. You would need to establish the actual power draw and cooling requirements before assuming a site can accommodate it.
Talk to the power provider about available capacity and delivery timelines. Then check rack space, electrical distribution and cooling against the equipment specifications. Being near a large power source does not necessarily mean the local connection can supply the load.
Reply to this answer
The listed configuration is more than a pile of accelerators. Each compute node supposedly has eight SXM4 A100s connected through NVSwitch, eight 200Gb/s network adapters and two 64-core CPUs. That could be useful for workloads spread across GPUs, but the listing alone does not tell you what network fabric, software or deployment work comes with it.
Check your model's memory and performance requirements against the 40GB GPUs. Also verify which interconnect components are included and whether your software supports the configuration you plan to run.
Reply to this answer
Used enterprise gear needs a careful inspection plan. Ask for an inventory, serial numbers, test results, condition, warranty terms and details of any missing parts. At this scale, even a modest number of failed components could mean a lot of troubleshooting.
Also get a clear agreement on how the equipment will be packed, shipped, delivered and accepted. A normal parcel delivery description is not enough to plan a move of hundreds of racks.
4b~jostlecap_739:
Would you test everything before accepting it?
_h3+bravik.7:
I'd agree on a practical test and acceptance process before buying, including what happens if nodes or components fail inspection.
Reply to this answer
Compare the total cost with the cloud capacity you actually use, not just the purchase price against a headline rental rate. Power, cooling, space, networking, maintenance and staff all matter, as does how consistently the GPUs will be busy.
On-prem might make sense for a sustained workload or for keeping data and infrastructure under your control. The listing by itself does not show what the savings or payback period would be.
Reply to this answer
Add your answer
Answers are reviewed before publication.
a.476!tumblegrommet:
And get the numbers for the full setup, not just the GPU power ratings. The rest of the rack and the cooling system consume power too.