I'm looking at a pod whose average CPU usage is well below its limit, but it still shows throttling. How can that happen, and what should I check?
Why would a pod at 40% CPU still get throttled?
Asked by
a.73~spareladle
Answers
CPU limits are enforced as a time budget within a scheduling period, not against the average shown on a dashboard. A short burst can use up that budget even if the container is mostly idle afterward.
For example, with a two-CPU limit and a 100 ms period, the budget is 200 ms of CPU time. Four threads running on four cores can consume it in about 50 ms of wall time. The container can then be throttled until the quota is replenished. A graph averaged over a longer interval can hide that burst.
Reply to this answer
Check the container's cgroup cpu.stat, especially nr_throttled and throttled_usec. Look at how those counters change over time and compare them with latency and short-interval CPU usage, rather than relying on one averaged usage graph.
Throttling is not the only reason a workload can wait for CPU. CPU pressure in cpu.pressure can help show contention, while hypervisor steal time can point to time the virtual machine didn't get from the host. Those are different signals from quota throttling.
Reply to this answer
For a latency-sensitive service, one option is to remove the CPU limit and keep a realistic CPU request. That lets it burst without hitting its own quota, but it also removes that cap, so it can use more node CPU under load and contend with other workloads.
Check the tradeoff for your cluster, including the pod's QoS class, before making that change. A request is not a hard maximum.
Reply to this answer
Removing limits isn't the only answer. If you need a cap, load test the service and size the request and limit for the expected load with some headroom. Then watch both throttling and latency under bursty traffic. Other workloads competing for CPU can still cause delays even when the pod isn't hitting its own quota.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Rivlet.5x:
With only two CPU-bound threads, that same budget would take roughly the whole 100 ms period to consume, assuming both run continuously. Actual scheduling can make the timing less tidy.