How are you tracking GPU costs by Kubernetes workload?

OpenCost is a solid starting point, but it helps to separate two kinds of cost. Allocated cost can be calculated from GPU requests and the node or GPU hourly price, then attributed through namespace, workload-owner, and job labels. Used cost should combine GPU utilization and memory metrics from NVIDIA DCGM Exporter with pod metadata from kube-state-metrics. Use rolling windows instead of instant readings because batch and inference workloads can be bursty. Also track pending GPU requests separately from GPUs that are allocated but mostly idle. For MIG, price the individual profile or slice rather than charging for a whole GPU, and be careful with time-sliced devices because utilization may not map neatly to one workload. A practical starting stack is OpenCost, kube-state-metrics, DCGM Exporter, Prometheus, and recording rules for requested GPU-hours, utilized GPU-hours, and idle-cost percentage.

We use OpenCost with a custom exporter that reads GPU metrics directly from the node, since nvidia-smi isn’t consistently available inside every pod. The exporter maps the data back to namespaces and workload labels in Prometheus, with Grafana for visualization. For idle capacity, we compare requested resources with actual utilization over a rolling period and flag GPUs below roughly 15% utilization for more than an hour. It’s a bit custom, but it gives us useful results.

We solved most of this by connecting utilization metrics to namespace and workload labels instead of looking only at allocation. NVIDIA DCGM Exporter and Prometheus provided the utilization data, and a simple Grafana dashboard let us compare cost with actual usage. It quickly exposed several jobs that kept GPUs reserved long after they stopped using them. That ended up being our biggest source of idle spend.

A lightweight alternative worth evaluating is Burn. It may be a better fit if you want basic cost visibility without installing an agent in the cluster.