I've been facing some issues with Karpenter lately. It seems that when subnet tags are either accidentally removed or mismatched with the NodePool subnetSelector config, nodes are being disrupted. Karpenter is unable to find matching subnets and ends up deleting nodes, which leaves my pods in a pending state without replacing those nodes. It looks like this is happening due to the expireAfter setting, but there's no safety feature to check whether nodes can be provisioned safely post-deletion. Does anyone have suggestions on how to implement Karpenter more safely to avoid disrupting existing nodes when new ones can't be provisioned due to these subnet mismatches?
2 Answers
Have you considered adding tags back to your subnets? I did that once and it fixed my problems at the time. However, the real issue seems to be about implementing Karpenter more safely. You definitely want to ensure that nodes aren’t forcefully disrupted, especially when your pods are left hanging in a pending state. Maybe looking into a better strategy for tagging or utilizing PDBs could help you out.
It sounds like you're dealing with a real headache! Karpenter does respect Pod Disruption Budgets (PDBs) even during node drains, so if you think your PDBs are being ignored, you might want to file a bug report. Make sure you have PDBs set up because without them, Karpenter might remove nodes without any safety net. Just a heads-up, there are a few scenarios where PDBs can be bypassed, like external disruptions or unhealthy nodes with auto-repair enabled.
Thanks for the tip! Yes, tagging the subnets worked temporarily, but my main concern is about a more robust way to manage Karpenter to prevent forced disruptions. Any more strategies or best practices you think I should explore?