Hey folks! I've been working on our Azure Kubernetes Services (AKS) infrastructure, and after resizing our node pools, I've noticed that the load distribution seems really imbalanced. The Kubernetes scheduler appears to be skewed—one of my nodes is running at 99% memory allocation, while others are under 50%. I'm used to how VMware's vCenter handles this sort of thing by auto-rebalancing loads to keep hosts from being overloaded. Is there a similar method or tool in Kubernetes that can help me achieve a more even load distribution across my worker nodes? I'm especially concerned because high memory usage on one node is causing some services to fail. I'd love to hear your insights or suggestions!
5 Answers
Make sure the memory requests for your pods are aligned with the actual memory they need. The scheduler considers these requests when deciding where to place your pods, not just the current usage. So if your requests don't reflect the reality, it could lead to these imbalances you're seeing.
Yep, you've got the right idea. Initialization happens at the request level, but if a pod goes over the limit, it gets killed, which could definitely cause unexpected failures.
It's all about sizing your pods correctly. Once you have suitable requests and limits in place, the default scheduler should pretty much handle load distribution effectively. Just make sure to enforce policies that require all pods to have these settings. You could also consider using a Vertical Pod Autoscaler (VPA) to help fine-tune your requests as you scale your workload, but keep in mind that doing so might make you a bit less efficient on resources. It's all about finding that balance!
If you're not already cycling your pods or nodes often, you might want to consider a descheduler. It can help to better manage how your resources are utilized and keep things balanced.
Remember, when we talk about "load," we're generally referring to the requests metrics. The load balancer does round robin, but the scheduler has its own mechanics. How you right-size everything can be a bit complex—consider looking into tools like cast.ai for help with this.
If you aren't using them already, see if you're spreading your pods across nodes with topology constraints and a low max skew. Getting your pods evenly distributed can be a really quick fix for your issue!
We think we set them right, like 800Mi for requests and 1500Mi for limits. So while I get that request sets the stage for the scheduler, do you think we could be using them incorrectly? Like is the scheduler going to kill a pod that exceeds its limit of 1500Mi?