I'm working on a project that needs to access Nvidia MiG resources from a K8S node. However, I don't have the budget for an actual GPU (around 20-30k). Is there a way to simulate or fake these resources? I considered creating a separate program for that purpose, but I'm curious if there's an easier solution already available.
4 Answers
An alternative approach could be to spin up a Vast.ai instance with a GPU. It's a more accessible way to get the resources you need without the hefty upfront cost.
Before you dive in, make sure you understand what information the program is looking for from the GPU. For instance, if it uses the `nvidia-smi` command, you could create a script with the same name on the host that outputs fake but plausible data. But be prepared for a bit of reverse engineering to figure everything out!
You might want to check out the Fake GPU Operator project. It offers a way to simulate GPU resources and it should support some MiG features too. It's a good starting point for your needs!
Thanks for the tip!
Tricking Kubernetes into recognizing fake MIG resources isn't straightforward since it relies on the NVIDIA device plugin for accurate info. If you're looking to simulate GPUs or MIGs without actual hardware, the Fake GPU Operator is a solid choice. It can set up a dummy device plugin that reports fake GPU and basic MIG-style resources. Just keep in mind, anything that tries to use CUDA or NVML won't work, but for showing resources, it's great!
Got it! It only needs to deploy a pod that's consuming node resources, and the algorithm picks which GPU resource to attach.