Stateless workloads seem like an easier fit for spot, but I'm curious about databases and other stateful services. If you tried it and moved back to on-demand, what caused the trouble? If you made it work, what did your setup need?
What broke when you tried running stateful workloads on spot instances?
Asked by
Tivwick.6r
Answers
The node disappearing can be less painful than what happens next. Long-lived connections drop, clients reconnect, and failover or leader election takes time. If several services reconnect at once, the knock-on load can turn a single interruption into a wider disruption.
That makes on-demand capacity appealing for critical stateful components, even when spot works well for stateless workloads. The useful question is not just whether the data survives, but how the rest of the system behaves during recovery.
Reply to this answer
One proposed approach is to checkpoint the running process during the interruption warning and restore it elsewhere, aiming to preserve memory and connections. That sounds promising for avoiding reconnect storms, but it depends on whether the workload can actually be restored that way. A normal drain remains the fallback if checkpointing doesn't work.
Reply to this answer
The operational tradeoff matters too. A short interruption warning, disruption budgets, restart consistency, and draining behavior are all worth checking before moving a database onto spot. Even without data loss, managing those failure paths can eat into the value of the savings.
Reply to this answer
Add your answer
Answers are reviewed before publication.
m3__notary+later:
Leader elections and long-lived connections can be especially awkward together. A workload may survive the restart but still take too long to become useful again.