I'm looking to optimize the use of AWS GPU instances for training and inference with our custom models. I've been struggling to find a suitable instance type for our workload since I need flexibility similar to what services like Runpod provide. On Runpod, I can select the exact number of GPUs (from 1 to 10) along with options for CPUs, storage, and RAM. However, on AWS, the configurations feel very bundled. For instance, to run 8 T4 GPUs, I'm limited to the gp4dn.metal instance, which comes with 96 vCPUs that I don't actually need. I've hit my service quota and while I've requested an increase, I'm baffled by the lack of flexible configuration options, even for smaller GPUs. I'm open to paying a bit more than Runpod if I can get similar flexibility. Is there an explanation for this? What are my options for making the most out of AWS GPUs? Ideally, I need between 1-5 GPUs in parallel with VRAM ranging from 15 to 80GB for most scenarios.
2 Answers
In AWS, GPU instances are designed with fixed configurations, tying GPUs to specific CPU counts. You're correct that you can't rent GPUs independently. Instead of going for one large multi-GPU instance, you might consider launching multiple smaller instances, like using five g5.xlarge instances with one GPU each. This helps avoid wasting CPU resources while fully utilizing the GPUs you're paying for. You may also want to check out AWS ParallelCluster for managing clusters effectively.
The way AWS designs its instance types is mostly to ensure predictable performance and optimize network capabilities, rather than offering flexibility in configurations. The CPU-to-GPU ratio is set because many users need to push a lot of data through the GPUs, not just use VRAM for inference. While this might feel inefficient for lighter workloads, it helps AWS manage capacity planning better.
Your best bet is to consider g5 instances for smaller GPU numbers, or use multiple smaller instances for your workload instead of one large instance. Some setups involve using separate CPU instances for preprocessing while leaving the GPU nodes less complex. If you want flexibility similar to Runpod, AWS might not be the best fit for that style since they're really focused on reliability and systematic integration first, rather than purely efficiency.
Could you explain what you mean by "pushing data hard through the GPUs"? I’m curious about that. I’m open to the idea of using multiple smaller GPU instances as a workaround, but I’m surprised by the limited availability of single GPU options with varying VRAM options, like A40, A100, or RTX 5090. What are some other platforms similar to Runpod that you’d recommend?
I completely agree! I've noticed the same lack of single GPU machines with a variety of VRAM. It just seems like there should be more choices available. As for alternatives to Runpod, I've had some success with other platforms that offer flexible GPU configurations.
Thanks for the suggestion! I’ll look into ParallelCluster. Although, I still feel like the GPU options in g5 instances are limited. Is there a way to access different GPU types like A40, A100, or RTX 5090 in a single instance?