For months, I assumed our Kubernetes cost problem required better tooling. Then we discovered that cluster CPU utilization was only around 25% because teams had heavily padded their resource requests years ago. Since Karpenter treats those requests as real capacity needs, it keeps provisioning larger nodes and cannot consolidate them effectively.
The technical fix seems straightforward: collect better usage data, reduce unrealistic requests, and let the scheduler pack workloads more efficiently. The organizational problem is harder. Teams are understandably reluctant to lower requests because they may be blamed if latency increases or an outage follows.
How did you get engineering teams and management to approve smaller requests without turning the effort into a standoff?
5 Answers
Be careful with CPU limits. In many workloads, removing CPU limits prevents unnecessary throttling, while requests still provide the scheduling signal needed for bin packing. Memory is different: reducing memory requests too aggressively can lead to OOM kills, so treat it separately and validate changes with real usage data.
A useful way to get sign-off is to start with obvious, low-risk waste and use the savings as proof. We found services reserving several CPUs per pod while using a small fraction of that amount, which added up to thousands of dollars each month. Make changes gradually, monitor latency and errors, and publish before-and-after results. That turns the discussion from 'please accept more risk' into a measurable capacity-management project.
The long-term answer is incentives and ownership. Show management the waste in concrete terms, then make teams accountable for reviewing their resource usage. If a team is charged internally for the capacity it reserves, or its service cost is visible to the people operating it, inflated requests become much harder to ignore. Also make it clear that over-provisioning can be a deliberate reliability trade-off, not automatically bad engineering.
Run the Vertical Pod Autoscaler in recommendation-only mode for a few weeks or months first. Use the observed data to propose new requests, review them with each team, and apply the changes manually rather than letting automation modify production immediately. That gives everyone evidence and a chance to catch unusual traffic patterns.
Do not assume every large request is careless padding. At one large company, shrinking reservations toward the 98th percentile caused outages because teams were compensating for noisy neighbors, network interrupt issues, logging sidecars, and bursty workloads. We eventually had to improve node isolation and performance first, then make a reliability promise about what a requested amount of CPU would provide. Workloads that occasionally burst could be moved to less-contended nodes instead of forcing every replica to reserve its peak.
The important distinction is that removing limits does not remove requests. The scheduler and Karpenter still need realistic requests to place workloads efficiently.