How do you track GPU cost by workload in Kubernetes?

There are also lightweight cost-monitoring options that don’t require installing a full agent on the cluster. They can be useful for a quick first pass, but make sure the tool can attribute usage to workload owners and account for MIG or shared GPU configurations. Otherwise, the numbers may look precise while still assigning the wrong cost.

The biggest improvement for us was collecting utilization data instead of looking only at allocation. DCGM Exporter and Prometheus gave us the metrics, and a Grafana dashboard grouped by namespace and workload let us compare cost with actual usage. That exposed several batch jobs holding GPUs for much longer than necessary, which accounted for most of our idle spend.

It helps to define two separate cost views: allocated cost, based on GPU requests multiplied by the node or accelerator price, and used cost, based on utilization and memory metrics. OpenCost, kube-state-metrics, and NVIDIA DCGM Exporter provide a practical foundation. Attribute Jobs through their Job or CronJob owners rather than ephemeral pod names, and distinguish pending GPU requests from GPUs that are assigned but mostly idle. For MIG, calculate cost by profile or slice, and be careful with time slicing because utilization may not map cleanly to one workload.

OpenCost is a solid starting point. We added a custom exporter that reads GPU metrics directly from the nodes, since tools like nvidia-smi aren’t always available inside application containers. Prometheus can then join those metrics with namespace and workload labels, and Grafana works well for visualization. For idle capacity, we compare requested resources with actual utilization over a rolling window and flag GPUs below about 15% utilization for more than an hour.