How can I protect my batch jobs from being disrupted by Karpenter's node consolidation?
Asked by
TechNinja42
I'm looking for effective ways to ensure that Karpenter doesn't interfere with my long-running or critical batch jobs during node consolidation in my Amazon EKS cluster. Karpenter's consolidation feature aims to cut down on costs by terminating underutilized nodes. However, if not properly set up, it can unintentionally evict active pods, including vital batch workloads. I found that using the custom `do_not_disrupt: "true"` annotation is a smart move to prevent this disruption, but I'm open to any additional suggestions or best practices for safeguarding my compute-intensive tasks, especially in scenarios like data processing pipelines or ML training. Anyone got tips?
Answers
Great insights! It's nice to see a topic like this instead of the usual job advice threads. I think your suggestion about using the `do_not_disrupt: "true"` annotation is spot on. It allows for fine-tuning which pods to prioritize during consolidation.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions