I'm running an on-premises Rancher/RKE1 Kubernetes cluster and need advice on node-level resource management. Occasionally, a pod uses nearly all the CPU or memory on its node. I understand that resource requests influence scheduling, while limits restrict how much a container can consume. Should requests and limits usually be set relatively close together to reduce the risk of one workload overwhelming a node?
Does Kubernetes automatically rebalance or move pods when a node becomes heavily utilized, or would that require an add-on such as the descheduler? I'm cautious about eviction because the cluster includes critical StatefulSets where disruption could affect quorum, stateful applications with slow persistent-volume recovery, and single-replica Deployments such as an FTP service where restarting the pod would cause downtime.
Would a combination of accurate requests and limits, taints and tolerations, node affinity, and PodDisruptionBudgets be the right approach? I'm also considering dedicating and labeling specific nodes for critical or single-instance workloads, then tainting those nodes so unrelated workloads cannot run there. Ideally, automated rebalancing should not evict or move these protected workloads. What approaches work well in production RKE1 or on-premises clusters?
4 Answers
Accurate resource requests are the foundation. Set requests based on realistic usage so the scheduler reserves enough capacity, and set limits when you need a hard ceiling—especially for memory, since exceeding a memory limit can lead to an OOM kill. CPU limits can also cause throttling, so they should not be applied blindly. Requests and limits do not have to be identical, but a very large gap allows workloads to burst into spare capacity and can make contention more likely. Leave headroom on every node rather than scheduling right up to its allocatable capacity.
For critical workloads, dedicate nodes using labels, taints, node affinity or nodeSelector, and matching tolerations. This prevents ordinary workloads from landing there, but a toleration alone does not force a pod onto those nodes—use required node affinity or a similarly strict placement rule as well. You can also use separate node pools and reserve capacity for system components. This reduces accidental contention, although it does not make a node immune to kubelet pressure eviction or hardware failure.
Be very cautious with a descheduler in this situation. It can improve distribution for suitable stateless workloads, but it is not a substitute for capacity planning, correct requests, or application redundancy. Exclude namespaces, nodes, or workloads that must not be moved, and test every policy in a non-production cluster first. A single-replica FTP service cannot be made highly available through scheduling alone; avoiding downtime requires application-level redundancy or accepting a maintenance window.
Kubernetes does not normally rebalance running pods just because a node becomes busier. The scheduler places pods when they are created or rescheduled, but it does not live-migrate them. Under resource pressure, kubelet may evict pods, especially when memory or disk becomes critically low, but that is an emergency mechanism rather than planned workload balancing. A descheduler can voluntarily evict pods according to configured policies, after which the scheduler places replacements, but it should be used carefully for stateful or single-replica applications.
That makes sense. I’m mainly trying to prevent an automated balancing policy from evicting critical workloads. Would tainted, labeled dedicated nodes plus matching tolerations and affinity be enough to keep those workloads isolated?
A PodDisruptionBudget helps limit voluntary disruptions, but it is not an absolute eviction shield. It generally protects availability during coordinated evictions, while involuntary events such as severe memory pressure, node loss, or a process being OOM-killed can still happen. For StatefulSets, maintain sufficient replicas and quorum, use appropriate anti-affinity, and ensure the storage and application can tolerate a restart.