I'm working with Karpenter to manage a separate node pool that only has "fast" nodes dedicated to one deployment. Currently, Karpenter is creating just a single node for all replicas, which is cost-effective, but I'm looking to improve resilience by distributing those pods across multiple nodes. I've tried using pod anti-affinity rules to ensure that replicas don't end up on the same node. I'm aware of topology spread constraints, but I think that if Karpenter chooses to initiate a single node, all pods will still be concentrated there. I'm considering limiting the available node sizes in the pool to enforce pod distribution, but this feels like a workaround and might hinder scalability if Horizontal Pod Autoscaler kicks in. Am I missing a better approach?
2 Answers
Don't forget to explore topology spread constraints! They might help you achieve the distribution you’re aiming for. Just remember that if Karpenter can fit all your pods on one node, it might still choose to do that.
You might want to look into switching from `preferredDuringScheduling` to `requiredDuringScheduling`. This way, you can enforce that pods need to be spread out across nodes, instead of just preferring that they be separate. It'll make sure no two pods end up on the same node regardless of how Karpenter schedules them.
That makes sense, but keep in mind that this only ensures each pod is on a separate node. There’s a balance to strike between resilience and overhead from daemonsets.
I checked out topology spread constraints, but it seems like Karpenter still won’t start additional nodes if it can fit all the pods on one.