I have a dedicated on-prem node with about 1 TB of RAM. A controller starts jobs that run for only one to three seconds, but new pods are staying Pending when there are roughly 30 active and 220 completed pods.
I want to scale toward 1,000 burst pods. Should I change the kubelet pod limit or cleanup settings, and what else should I check?
Start with the Pending pod events rather than assuming the completed pods are blocking it. Describe a few Pending pods and check whether they are waiting for scheduling, sandbox creation, networking, or something else. Those point to different fixes.
Also check the node's allocatable pod count and its current kubelet maxPods setting. Raising that limit may remove one constraint, but it won't fix an IP shortage or a component that can't keep up with the creation rate. Lots of RAM doesn't tell you how many pods the node can start and stop quickly.
That number near 250 makes the pod network worth a look. Check the node's pod subnet and the address capacity and allocation rules of your network plugin. A subnet around a /24 can be a constraint, though the exact usable count depends on the setup.
Don't just raise maxPods to 1,000 without checking that the network can actually assign addresses at that scale.
If these are Kubernetes Jobs, you can set spec.ttlSecondsAfterFinished to 0 to make a finished Job eligible for immediate cleanup. The TTL controller deletes the Job and its dependent objects, so this isn't a kubelet setting and deletion isn't necessarily instantaneous.
Make sure your controller records the result before cleanup if you need to track successes and failures. If you're creating standalone pods instead of Jobs, that Job TTL field won't clean them up.
For tasks that last a few seconds, a pod per task can spend a lot of effort on startup and cleanup relative to the work itself. If the application can change, consider a queue with longer-lived workers that pull multiple tasks.
You can still track completion and errors per task in the queue or controller. It changes the design, though, and may not fit if each task really needs its own pod.
9c.after.lunch:
Per-task success and failure tracking matters here, so the controller would need to retain those results with either design.
Answers are reviewed before publication.
_perhaps__later:
Completed Job objects and completed pods are different things. Check which objects are accumulating before deciding what needs cleaning up.