I'm running a Kubernetes operator built with Quarkus as a GraalVM native image and using the Java Operator SDK. It contains four controllers—Permission, Entitlement, PartyRole, and UserRole—and each polls on a timer. The container has a 500m CPU limit on cgroups v1 with kernel 4.15. Although average CPU usage is only around 13% of the limit, the container still shows frequent CFS throttling, with the throttled-period rate around 0.47 over 10 minutes.
I reduced concurrent reconciliation threads from the default of 50 per pool to 4, which reduced the total thread count from roughly 130 to 39 and improved throttling somewhat. I also added a semaphore so the four controllers do not poll simultaneously, but that only slightly reduced CPU use and did not significantly change the throttling pattern. I'm now testing one reconciliation thread and one workflow thread, along with 15-minute polling intervals and up to 10 seconds of polling jitter.
One controller, Permission, appears to issue hundreds of requests to an external downstream service almost at once during its polling cycle. I suspect these short bursts may be exhausting the CPU quota even though the longer-term average is low, so I'm considering adding delays or otherwise rate-limiting those requests.
Is this primarily a burst and scheduling problem rather than a lack of total CPU capacity? Are there JVM, native-image, Quarkus, Vert.x, or cgroup settings that could help reduce throttling, beyond continually lowering concurrency?
3 Answers
The combination of cgroups v1 and a 4.15 kernel is worth investigating. Older CFS quota accounting could behave poorly for bursty workloads, particularly when a container has a tight CPU limit. If possible, compare the same workload on a newer kernel or cgroups v2.
Also test the operator with a higher CPU limit, or with only a CPU request if strict limiting is not required. If throttling largely disappears when the limit is raised, that confirms the issue is quota exhaustion rather than a permanently high CPU workload. Check the throttled-period counter alongside short-window CPU usage instead of relying only on a long average.
The four controllers are already running inside one Quarkus/JOSDK process, so splitting or consolidating them as services is not the key issue here. The important distinction is the burst of calls made by the Permission controller to the external service. Since the controllers are not communicating with one another over the network, changing the application architecture will not by itself address CFS throttling.
A single-process design with bounded concurrency, staggered timers, jitter, and backpressure is reasonable. Just make sure the downstream request executor is bounded too—limiting reconciliation threads alone will not help if each reconciliation can fan out into hundreds of simultaneous asynchronous requests.
The 13% average is probably hiding short CPU spikes. With a 500m limit and a 100 ms CFS period, the container can receive roughly 50 ms of CPU time per period. A burst—such as one controller launching hundreds of downstream calls, or native-image and garbage-collection activity—can consume that allowance quickly. The container is then throttled even though a 10-minute average looks very low.
Reducing thread counts may help by reducing contention and context switching, but it does not remove the underlying quota boundary. Smoothing the Permission controller's work with batching, rate limiting, bounded queues, or deliberate spacing between requests is likely more effective than simply setting every pool to one thread. Monitor CPU usage at a much shorter interval as well, since long-range averages can conceal these bursts.
That matches the setup: it is one native Quarkus operator, and the large burst comes from one controller's calls to an external service. I'll focus on bounding and spacing that fan-out, then compare short-interval CPU metrics with the throttling counter.