I'm running a Kubernetes operator built with Java, Quarkus, the Java Operator SDK, and a native GraalVM image. The pod has a 300m CPU limit and 256Mi memory limit. Despite average CPU usage being only about 13% of the limit, container_cpu_cfs_throttled_periods_total stays consistently around 1.2–1.7 throttled periods per second over 24 hours, without any obvious relationship to traffic or polling activity.
Metrics show roughly 129–131 live JVM threads, with occasional peaks around 196. The Vert.x worker pool originally reported about 199 idle threads and only one active thread, so I suspected many brief thread wakeups were exhausting the CFS quota. With a 300m limit, the container receives only about 30ms of CPU time per 100ms CFS period.
I tried setting -XX:ActiveProcessorCount=1, but the live thread count barely changed and throttling actually increased. It also caused Java Operator SDK health-check instability, so I reverted it. Limiting ParallelGCThreads and ForkJoinPool parallelism in a local JVM test also had no effect.
I then changed the Vert.x worker pool through a VertxOptionsCustomizer, reducing it from the default size to eight threads. This worked for that pool—the metrics changed to roughly six idle and two active workers—but throttling and total live threads stayed essentially unchanged. The original worker pool therefore may not be the real source of the behavior.
How should I identify where the remaining native-image threads come from on a distroless container without shell access or ephemeral-container support? Could they be from the operator SDK, reconciler executors, garbage collection, or other runtime pools? Is 130 live threads inherently problematic for a 300m pod, and are JVM flags such as ActiveProcessorCount expected to behave differently in a GraalVM native image? Would increasing or removing the CPU limit be the pragmatic solution, or is there a better way to diagnose and tune the bursty throttling?
4 Answers
For a latency-sensitive operator, I’d test with a substantially higher CPU limit or no CPU limit while keeping the request appropriate for scheduling. If throttling disappears and responsiveness improves, that confirms the quota is contributing, even if it doesn’t identify the thread source.
Removing the limit is an operational choice, though: it allows bursts and can make noisy-neighbor behavior or node-level contention more relevant. Increasing it gradually is a safer diagnostic than immediately redesigning every executor.
The low average is not contradictory. With a 300m limit, Linux CFS generally gives the container about 30ms of aggregate CPU time in each 100ms period. If a burst wakes many threads at once, that quota can be consumed almost immediately, after which the container is throttled until the period resets. A scrape showing 13% average usage can therefore hide repeated short freezes.
Check throttled seconds as well as throttled periods, and compare both with a higher CPU limit. Also look at runnable or queued work if you can expose it. The important distinction is burstiness and latency, not just total CPU consumed.
Reducing the Vert.x pool was still a sensible experiment, but since the pool metrics changed while throttling and live-thread counts did not, it probably isn’t the main source. A live-thread count by itself also doesn’t prove that all 130 threads are waking or consuming CPU; many could be parked permanently.
I’d focus on finding runnable bursts and the actual executor owners rather than treating the total thread count as the culprit. Native-image build-time defaults and framework-created executors may not respond to the same runtime flags as a HotSpot JVM, so verify each setting against the specific Quarkus and GraalVM versions instead of assuming ActiveProcessorCount controls every pool.
Since the image is distroless, add observability to the application rather than relying on shell tools. Depending on what the native runtime exposes, useful options include periodically recording thread names and states, exporting executor queue and active-count metrics, and adding startup diagnostics for every executor you create. You can also inspect the process from a debug-capable sidecar or node-level tooling if your cluster security policy permits it.
The two named Vert.x pools won’t account for every thread. The operator SDK, HTTP client, timers, event loops, native runtime, and GC may use separate executors or threads that don’t appear in those metrics. A thread dump or equivalent native diagnostic is much more useful than the aggregate live-thread gauge.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures