For months, I assumed our Kubernetes cost problem required more sophisticated tooling. Instead, we found that our clusters were averaging only about 25% CPU utilization because teams had padded their resource requests years ago. Karpenter treats those requests as real capacity requirements, so it provisions larger nodes and cannot consolidate them effectively.
The technical fix seems straightforward: measure actual usage, reduce inflated requests, and let the bin packer work with better input. The organizational problem is harder because the team that lowers a request may feel responsible when the next latency incident happens.
What process or incentives helped you get approval for reducing requests without turning it into a standoff?
5 Answers
Be careful about treating every oversized request as waste. At one large environment, aggressively shrinking reservations caused outages because teams had padded for noisy neighbors, network interrupts, logging sidecars, and bursty workloads. We eventually tuned the underlying cluster issues and moved consistently overusing pods to less-contended machines before making stronger capacity guarantees.
Removing CPU limits can help avoid throttling and makes teams less worried about being blamed for changing a limit. After that, trim CPU requests based on observed usage so the scheduler and node provisioner can pack workloads more efficiently. I would be much more conservative with memory requests, since cutting those too far can lead to OOM kills.
Use VPA in recommendation-only mode for a few weeks or months and review the results with each team. Apply the changes manually rather than letting it rewrite production workloads automatically. That gives teams evidence and a chance to account for traffic spikes before changing requests.
The lasting fix is to make resource usage visible and give teams an incentive to manage it. Show management how much capacity each service reserves versus what it actually uses, or allocate infrastructure costs back to the owning teams. People naturally over-provision when outages are punished but idle capacity is free, so the organization has to make both sides of that trade-off explicit.
That makes sense. The goal is not to force every service to run at minimum capacity, but to make the cost and reliability trade-off visible to the people choosing the padding.
A practical way to get approval is to frame this as a measured cost-reduction project with guardrails. Start with low-risk services, use historical percentiles instead of arbitrary cuts, set rollback thresholds for latency and errors, and publish the savings. Once teams see that the process protects their service, it is much easier to expand it without making anyone personally own a risky number.
That approach worked for CPU, but I would still validate it against tail latency and burst behavior rather than relying only on average utilization.