I'm running an on-premises Rancher/RKE1 Kubernetes cluster and occasionally see a pod consume nearly all available CPU or memory on its node. I understand that resource requests influence scheduling while limits cap container usage, but I'm unsure how closely they should be set to prevent one workload from affecting other pods.
Does Kubernetes automatically rebalance or move pods when a node becomes heavily utilized, or would I need a tool such as the descheduler? I'm particularly cautious because some workloads are critical StatefulSets where eviction could affect quorum, while others use persistent storage and take a long time to restart. I also have single-replica Deployments, such as an FTP service, where eviction would cause downtime.
Would a combination of realistic requests and limits, taints and tolerations, node labels and affinity, PodDisruptionBudgets, and dedicated nodes be the right way to isolate these workloads? Ideally, I'd like critical workloads on specifically tainted and labeled nodes, protected from voluntary descheduling while still allowing normal scheduling and node-pressure protection to work.
3 Answers
If you use a descheduler, configure exclusions for the workloads that cannot tolerate voluntary eviction and consider priority classes, namespace or label-based filters, and PDB-aware strategies. Keep in mind that PDBs only constrain voluntary evictions and may still permit disruption when the budget allows it.
Also monitor actual CPU and memory usage, pod restart times, storage attach and mount delays, and node allocatable capacity. If one pod can legitimately consume nearly an entire node, assigning it an appropriate request and placing it on a dedicated node is generally more predictable than relying on later rebalancing. For workloads that must survive node loss, run multiple replicas across failure domains rather than depending on a scheduler moving one replica after the problem occurs.
Kubernetes does not normally rebalance running pods just because one node has become busier than another. The scheduler places pods when they are first created; moving an existing pod requires deleting and recreating it. Node-pressure eviction is a separate safety mechanism that can evict pods when a node is running critically short of memory, disk, or other resources, but it is not a general load-balancing feature.
Start with accurate resource requests, since those are what the scheduler uses for placement. Set limits where you need an upper bound, and avoid allowing a pod to consume unbounded memory. Requests and limits do not always have to be identical, but a large gap can result in burstable workloads competing for spare capacity. Use namespace LimitRanges and resource quotas to prevent accidental omissions, and leave sufficient headroom on every node.
For critical or slow-to-restart workloads, dedicated nodes are a reasonable design. Taint those nodes so ordinary pods cannot land there, add a matching toleration to the intended workloads, and use required node affinity or nodeSelector to ensure they are scheduled only on the labeled nodes. This is placement isolation, not an absolute guarantee against every eviction: hard node-pressure eviction, node failure, maintenance, and manual deletion can still affect pods.
PDBs help limit voluntary disruptions, but they do not prevent all evictions and they cannot make a single-replica application highly available. For a single-instance FTP service, the real options are accepting downtime, adding redundancy, or using an architecture that can fail over. For StatefulSets, make sure the replica count, quorum rules, storage behavior, and disruption budget are compatible before allowing automated eviction.
The descheduler can usually be configured with filters and strategies so selected namespaces, pods, or nodes are excluded, but that configuration should be tested carefully. It is safer to treat descheduling as an optimization tool rather than a protection mechanism.
So tainted, labeled nodes can be reserved for critical workloads, as long as those pods have the matching toleration and node affinity, right?