I manage on-premises clusters and have a dedicated node with roughly 1 TB of RAM. A controller pod creates Kubernetes Jobs for very short tasks that usually run for 1–3 seconds. The node currently has about 30 active pods and more than 220 completed ones, while over 100 new pods remain Pending. I initially assumed the kubelet was falling behind because completed pods were not being cleaned up. Is there a kubelet setting that removes finished pods immediately? Would increasing the node limit to 1,000 pods simply result in more Pending pods? The API QPS and burst settings are still at their defaults. The current pod network range also only supports around 250 pod addresses, but the long-term idea was to expand it and run roughly 1,000 short-lived pods on this node.
4 Answers
Completed pods generally do not consume scheduling capacity once they have terminated, although the Job and Pod API objects can still create a lot of control-plane and storage overhead. Configure the Job with `spec.ttlSecondsAfterFinished: 0` if you want the finished Job cleaned up immediately, or use a small positive TTL if you need a short period for inspection. That setting belongs to the Job resource rather than the kubelet.
A limit around 250 strongly suggests the per-node pod network range is the immediate ceiling. A `/24` typically provides only about 253 usable pod addresses, depending on the networking implementation. Expanding the CIDR may remove that particular limit, but it will not make the node capable of efficiently handling 1,000 rapidly created and deleted pods.
Also check the CNI plugin and its configured per-node address pools, not just the cluster CIDR. The networking layer may impose a lower limit than the Kubernetes pod setting.
The workload pattern is probably the bigger issue. Creating one pod for a 1–3 second unit of work adds substantial startup, networking, scheduling, and cleanup overhead. A queue with long-running workers can process many tasks per pod, and an autoscaler such as KEDA can add workers as the backlog grows. You can still track success and failure using task IDs and application-level status rather than making every task a separate pod.
If each task truly must be isolated, consider distributing the work across multiple smaller nodes or using a workflow system that chains several steps within fewer pods. That will usually be more reliable than forcing 1,000 short-lived pods onto a single node.
Raising `maxPods` can absolutely allow more pods to exist, but it does not fix the underlying bottleneck. Kubernetes guidance commonly uses roughly 110 pods per node, and thousands of short-lived pods on one node can overwhelm the scheduler, API server, kubelet, CNI, container runtime, and garbage collection. More memory alone does not eliminate those costs.
The Pending pods should be investigated separately. Check their events and scheduler messages for problems such as exhausted pod IPs, sandbox creation delays, volume or resource constraints, or general scheduling throughput.