I'm a bit confused about how Kubernetes deals with memory limits relative to available resources. Let's say you have a single node with 8 GiB of memory and you want to run 3 pods. Each pod can spike to 3 GiB, but they never spike simultaneously, so 8 GiB total should be sufficient. If you configure each pod with resource requests of 1 GiB and limits of 3 GiB, the sum of requests equals 3 GiB, which is okay. However, the sum of limits totals 9 GiB, which exceeds the node's capacity. My questions are: Is this scenario allowed by Kubernetes? Will the scheduler or ResourceQuota reject it because the total `limits.memory` exceeds the available 8 GiB? Also, if a ResourceQuota in my namespace sets a hard limit of 8 GiB for memory, would the pods fail to start due to the total limits being 9 GiB? I want to clarify whether exceeding total `limits.memory` against physical or quota hard limits is permissible or if it gets blocked.
1 Answer
In Kubernetes, it's possible for the sum of limits to exceed the physical resources of the nodes. The kube-scheduler uses the requests to decide where to place pods, while limits are enforced by the kubelet. If the total limits go over the available memory and an actual spike occurs, the kubelet might get OOM killed by the OS before it can evict any pods. So, it's quite risky to exceed actual node capacity with limits.
Yeah, it's especially tricky since the kubelet may not handle exceeding limits gracefully. I’ve seen pods getting OOMKilled when memory spikes unexpectedly, which leads to a horrible time trying to recover.