I was curious about the reasons for draining a node in a Kubernetes cluster before upgrading it. What kind of issues could arise if a node isn't drained first? For instance, if a node suddenly goes down, how does Kubernetes handle pod eviction in that scenario?
4 Answers
Yeah, if the node goes down suddenly, you'd need to manually delete the pods. That's why draining is safer—having enough replicas and using affinity rules can also help prevent multiple pods of the same workload from ending up on the same node.
If a node crashes, Kubernetes loses track of the pods running there and marks them as unknown, waiting for the node to return. Draining or deleting the node in Kubernetes can help to reschedule those pods—draining is the gentler option that respects disruption budgets.
If a node goes down and doesn’t come back, you might end up with pods stuck in an "Unknown" state, which can require manual force eviction or deletion. Draining the node allows Kubernetes to quickly provision pods on another node, minimizing downtime. Also, don't forget to cordon the node before you drain it!
It really depends on what you're running on that node. For example, I've seen rook-ceph nodes fail if the kubelet restarts unexpectedly. It's always a good practice to have a strategy for recovery depending on what you're hosting.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures