Why Is My Quarkus Native Operator Still CFS-Throttled Despite Low Average CPU Usage?

Reducing thread counts can help, but it does not create more CPU quota. Many runnable threads competing for a small fixed quota can add context switching and make bursts worse, so getting down to a small, bounded pool is reasonable. However, setting every pool to one may simply serialize useful work without addressing the downstream request burst.

I'd inspect the actual executor configuration used by JOSDK, Quarkus, and Vert.x rather than assuming the visible reconciliation setting controls every thread. Check event-loop versus worker-pool usage, queue lengths, GC pauses, and native-image runtime threads. A rate limiter or bounded worker queue around the bulk request loop is likely more effective than a global semaphore around all controllers.

This is one operator process with four controllers, not four microservices, so splitting or consolidating services is unlikely to explain the throttling. The important detail is the burst from the Permission controller and the external calls it makes. Keep the controllers independent if that matches the design, but avoid firing the entire batch at once.

Try a small request concurrency limit, per-request backoff, and randomized scheduling so a poll cycle does not create one large runnable burst. Compare throttled periods, CPU usage sampled every second or less, request latency, and reconciliation completion time before and after. If raising the CPU limit makes the throttling disappear immediately, that confirms the issue is quota-period burstiness rather than the operator's long-term CPU demand.

The low average is not contradictory. With a 500m CPU limit, CFS may give the container roughly 50 ms of CPU time per 100 ms period. A short burst—such as one controller launching hundreds of downstream requests, startup activity, or GC/native-runtime work—can consume that quota quickly. The container can then be throttled for the rest of the period even though its 10-minute average remains low.

Pacing the downstream requests, adding bounded concurrency, and spreading poll cycles with jitter are good experiments. Also monitor CPU at a much finer resolution than a 10-minute average so you can correlate the spikes with throttled periods. On an old cgroups-v1 setup with a 4.15 kernel, it is also worth checking for CFS accounting behavior or kernel-related over-throttling. If possible, test with a higher CPU limit or with only a CPU request to confirm whether the quota is the trigger.