How to Configure Karpenter to Wait Before Forcefully Terminating Pods?
Asked by
CuriousCoder42
I'm using Karpenter with an EKS setup that relies on EC2 spot instances. I have an application that requires a 30-second grace period before the pods can be terminated. I've set a lifecycle hook (preStop) which works fine when I manually drain nodes or delete pods. However, I'm encountering an issue where Karpenter forcefully evicts pods when it gets a spot interruption message via SQS. Although my application doesn't crash due to a configured pod disruption budget (PDB), I need to find a way to get Karpenter to respect that 30-second wait time before terminating the pods. Any suggestions?
Answers
Have you looked into the terminationGracePeriod setting for your node pool? Apparently, Karpenter allows you to specify a grace period, and if you check out their documentation, it mentions how to set that up. It might help with your timing issue! Check out this link for more details: [Karpenter Node Pools](https://karpenter.sh/docs/concepts/nodepools/#spectemplatespecterminationgraceperiod).
Reply to this answer
Just a thought – did you ensure that you set both the preStop hook and the terminationGracePeriodSeconds? I’ve seen some users overlook that, thinking the preStop was enough. It might be worth double-checking.
CuriousCoder42: Yes, I set both to 100 seconds. The manual drains work fine, but when spot interruptions hit, it seems like the pods just get yanked out immediately.
Reply to this answer
Have you configured the AWS event rules for spot instance termination alerts when setting up Karpenter? Plus, setting up the interruption queue could be crucial; it might allow you to manage the evictions better.
Reply to this answer
Hey, since you're all about spot instances, you're probably saving a ton! If you haven't already checked it out, Rackspace Spot offers some neat features like pre-emption notifications and real-time capacity insights, which could help mitigate these issues. Just something to consider!
Reply to this answer
Add your answer
Answers are reviewed before publication.
AppEnthusiast86: Thanks for the tip! I thought that if I didn’t set it, it would just act like a normal drain without forceful terminations.