I'm trying to understand some unexpected node churn with Karpenter. Why would it consolidate a node that doesn't look underused, and what can I do to reduce unnecessary disruption?
Why is Karpenter consolidating nodes that seem fine?
Answers
Consolidation isn't just a judgment about how busy that one node looks. Karpenter also considers the rest of the cluster and whether workloads can be placed elsewhere. A node can look perfectly reasonable in isolation and still be a candidate for consolidation.
Check the placement options across the cluster when investigating churn, rather than looking only at the utilization of the node that was removed.
Reply to this answer
For workloads that move around a lot, take a look at consolidationPolicy and consolidateAfter. WhenEmptyOrUnderutilized allows consolidation beyond empty nodes, while a suitable delay can give short-lived changes time to settle.
For pods that really shouldn't be moved, consider the karpenter.sh/do-not-disrupt annotation. Use it selectively, since preventing disruption can also limit consolidation.
Reply to this answer
Separate pools for short-lived batch jobs and long-running workloads are worth considering. That can isolate some of the churn, though separate pools alone aren't a complete fix.
There's a tradeoff if you keep splitting pools: fewer workloads can share the available space, so bin packing may get worse. Stateful workloads can be especially awkward because restarts are costly, but they may also benefit from larger nodes with more storage throughput.
Reply to this answer
Also check the Kubernetes scheduler configuration. Its placement preferences can work against Karpenter's consolidation decisions and contribute to churn. Switching to a packing strategy such as MostUtilized isn't necessarily a cure either, since similar conflicts can still happen.
Reply to this answer
Check which Karpenter release you're running and its consolidation changes before tuning around this. Releases can change how aggressively consolidation behaves, so advice based on a different version may not fit yours.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Vexlin-5c:
So separating batch jobs might help, but putting every stateful workload in its own pool could waste capacity?
7d+papercradle_518:
Right. The pool boundaries affect what can be packed together, so it's worth weighing isolation against how well each pool uses its nodes.