I have a service running as one replica on a spot node. Is there a way to bring up a replacement before a planned drain or spot interruption without paying for a second replica all the time?
Can I drain nodes without downtime for a single-replica service?
Answers
You can make planned drains less disruptive with an opt-in controller that temporarily scales the workload from one replica to two. It needs to detect an impending disruption, prevent eviction of the old pod, wait for the replacement to be ready and receiving traffic, then allow the drain and restore the replica count.
That is best effort, not a zero-downtime guarantee. An interruption deadline may arrive before a new node starts, the image pulls, and the pod becomes ready. Sudden node failures also leave no time to surge. Make sure the detection covers the signals your node manager actually uses, whether that is a cordon, a taint, or a cloud interruption notice.
Reply to this answer
If zero downtime is a hard requirement, run at least two replicas and spread them across nodes. A single replica has no ready backup when its node fails unexpectedly, and a PodDisruptionBudget only controls voluntary disruptions. Even two replicas do not guarantee availability in every failure scenario.
For a development environment, accepting occasional disruption might be a reasonable cost tradeoff. That is a different target from high availability, though.
Reply to this answer
Be careful with a PodDisruptionBudget that blocks every eviction. It can also hold up upgrades and scale-downs if the surge controller fails or the replacement never becomes ready.
Give the process a deadline and a way to release the block if it cannot make progress. Monitor stuck drains too. Otherwise a mechanism intended to reduce downtime can leave nodes unable to drain.
Reply to this answer
Check whether the application can safely run two copies at once before trying a surge. One replica may be a cost setting, but it may also be an application constraint. Make the behavior opt-in for workloads that tolerate a brief overlap.
Reply to this answer
Also compare the cost of a more stable node with the cost and maintenance of a custom controller. If the service is small, the absolute price difference may be modest. An autoscaled node pool can provide replacement capacity, but it still takes time to provision a node and start the pod.
Reply to this answer
Add your answer
Answers are reviewed before publication.
b.619+parcelwhisk:
A spot termination handler can provide the warning and start a drain, but that alone does not create a ready replacement before the single existing pod is evicted.