Can I drain nodes without downtime for a single-replica service?

November 16, 2026 5 answers
Asked by _q26+nodepiston.7

I have a service running as one replica on a spot node. Is there a way to bring up a replacement before a planned drain or spot interruption without paying for a second replica all the time?

Answers

6e+buttoncase_841

You can make planned drains less disruptive with an opt-in controller that temporarily scales the workload from one replica to two. It needs to detect an impending disruption, prevent eviction of the old pod, wait for the replacement to be ready and receiving traffic, then allow the drain and restore the replica count.

That is best effort, not a zero-downtime guarantee. An interruption deadline may arrive before a new node starts, the image pulls, and the pod becomes ready. Sudden node failures also leave no time to surge. Make sure the detection covers the signals your node manager actually uses, whether that is a cordon, a taint, or a cloud interruption notice.

b.619+parcelwhisk:

A spot termination handler can provide the warning and start a drain, but that alone does not create a ready replacement before the single existing pod is evicted.

Reply to this answer
_p86!hingesieve.3

If zero downtime is a hard requirement, run at least two replicas and spread them across nodes. A single replica has no ready backup when its node fails unexpectedly, and a PodDisruptionBudget only controls voluntary disruptions. Even two replicas do not guarantee availability in every failure scenario.

For a development environment, accepting occasional disruption might be a reasonable cost tradeoff. That is a different target from high availability, though.

Reply to this answer
Tovren+6m

Be careful with a PodDisruptionBudget that blocks every eviction. It can also hold up upgrades and scale-downs if the surge controller fails or the replacement never becomes ready.

Give the process a deadline and a way to release the block if it cannot make progress. Monitor stuck drains too. Otherwise a mechanism intended to reduce downtime can leave nodes unable to drain.

Reply to this answer
r.582+bitgauge

Check whether the application can safely run two copies at once before trying a surge. One replica may be a cost setting, but it may also be an application constraint. Make the behavior opt-in for workloads that tolerate a brief overlap.

Reply to this answer
6n~baudspool_493

Also compare the cost of a more stable node with the cost and maintenance of a custom controller. If the service is small, the absolute price difference may be modest. An autoscaled node pool can provide replacement capacity, but it still takes time to provision a node and start the pod.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions