I'm looking for advice on managing a situation where a few pods are keeping entire nodes running in our Kubernetes cluster. From a FinOps perspective, it's frustrating to see a lot of nodes sitting at 20-30% utilization. While I initially felt optimistic about consolidating those nodes, I learned from the DevOps team that some pods are 'unevictable' due to factors like pod disruption budgets, local storage needs, strict affinities, or simply because there aren't other nodes that can host those pods. This means we have nodes that are mostly idle except for one or two pods. I'm trying to find effective strategies or proposals to present to the DevOps lead that go beyond just 'move the pods.' What are some common practices for dealing with this issue?
5 Answers
The crux of the issue often revolves around shared storage. If certain pods depend on local storage, like NVMe disks for databases, they can't just be moved. Shared storage can be a solution, but it brings its own set of challenges and costs.
If you're running your Kubernetes cluster on-premise, local storage could be a major blocker. High-performance distributed storage can make migrating stateful pods much easier. Tackling the issue of local storage will likely improve your situation significantly.
First, it's essential to check if maintaining those pods on their nodes is intentional. Ask the DevOps team if the node sizes are appropriate. Some reasons you listed don’t justify inaction; they help explain why the system is set up this way, so it's crucial to have that discussion.
Sometimes the best approach is to look at the bigger picture and see if other pods can be shifted around to free up nodes. Start strategizing ways to deal with those unevictable pods right from the get-go. It's all about optimizing the entire system.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures