Hey everyone! I've set up Karpenter in my EKS cluster and I'm currently running some load tests. I've got a horizontal autoscaler configured with a 2 CPU limit, and it's supposed to scale up to 3 pods at once. However, when I trigger a scale up, Karpenter creates 4 nodes, each with 4 VCPUs (using the c5a.xlarge instance type). Is this normal behavior?
Here are the resource settings I'm using:
resources {
limits = {
cpu = "2000m"
memory = "2048Mi"
}
requests = {
cpu = "1800m"
memory = "1800Mi"
}
}
And my Karpenter Helm configuration looks like this:
settings:
clusterName: ${cluster_name}
interruptionQueue: ${queue_name}
batchMaxDuration: 10s
batchIdleDuration: 5s
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: ${iam_role_arn}
controller:
resources:
requests:
cpu: "1"
memory: 1Gi
limits:
cpu: "1"
memory: 1Gi
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: karpenter.sh/nodepool
operator: DoesNotExist
- key: eks.amazonaws.com/nodegroup
operator: In
values:
- ${node_group_name}
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: "kubernetes.io/hostname"
I thought Karpenter would scale to 3 nodes to match the 3 pods, but even after setting batchIdleDuration and batchMaxDuration, the behavior hasn't changed. Should I expect fewer nodes that are more powerful instead? Thanks in advance for your insights!
2 Answers
You're not alone in this – many folks find that Karpenter's decisions can seem a bit off at first. It generally manages resources pretty intelligently, but definitely keep an eye on how vCPUs are getting allocated. It might help to check if you're running the metrics server to ensure accurate resource tracking – without that, Karpenter might be making decisions based on incomplete data! Also, if you're using varying node sizes (like small, medium, large), try to stick to a more consistent type to see if that stabilizes your node provisioning.
And as for your concern on pods - definitely make sure you're setting appropriate limits and requests to ensure everything gets managed smoothly!
It sounds like Karpenter is trying to ensure there's plenty of capacity for both your workload and itself. Each of your pods requests around 2/3 of a CPU and each node has more than that, which might be leading Karpenter to over-provision nodes. Rather than sticking to 3 nodes, it's likely that Karpenter is creating an additional node to accommodate its own controller resources and to ensure the pods are able to fit comfortably without being squeezed.
One approach you could take is to specify node requirements more explicitly, like preventing Karpenter from provisioning nodes with fewer vCPUs, which could keep your node count down to what you expect. You might want to dive into your nodeGroup settings since they can heavily influence Karpenter’s behavior. Check out the Karpenter docs for more details on configuring node pools effectively!
Thanks for the insight! I was wondering if there's a way to specify minimum node sizes that Karpenter can use. I’ll definitely have to adjust those configurations your way.
Good point about the metrics server! I had it set up before but didn’t think about it recently. I’ll get that verified!