I'm new to Kubernetes and trying to understand what happens when a worker node loses network connectivity because of a failure, while the machine and its processes remain running. Do the pods on that node continue operating? If Kubernetes later starts replacement pods elsewhere, how are changes made during the outage reconciled—are they overwritten, duplicated, or appended? I'd also appreciate guidance on the effects of network partitions involving services, storage, and stateful applications.
3 Answers
Kubernetes does not reconcile application data. It reconciles workload objects such as Deployments and StatefulSets. If the control plane creates a replacement pod elsewhere, the original process may still be running on the isolated node, so both could potentially write at the same time. A normal pod is not automatically fenced just because its pod object was deleted. That can create duplicate workers or conflicting writers. Storage behavior also varies: an emptyDir is local to each pod, local volumes usually prevent useful rescheduling, and ReadWriteOnce storage may block the replacement until the old attachment is detached or the old writer is fenced.
The final result depends on the database or storage system, not Kubernetes. A single-leader database generally uses quorum, leader election, and sometimes fencing to prevent split-brain writes. A multi-writer system may allow both sides to accept changes and later resolve conflicts according to its own rules. Some systems replicate and catch up after the connection returns, while others can lose or duplicate operations unless the application uses idempotency and durable queues. The key distinction is that restoring network connectivity does not automatically merge data; the storage layer must define how that works.
If the node itself is healthy, its existing pods can keep running, but they’re effectively isolated. They won’t receive new configuration or control-plane instructions, and they may lose access to services, other nodes, or network-backed storage. Kubernetes eventually marks the node unreachable and may evict managed pods, but this isn’t immediate. The timing depends on node monitoring, tolerations, eviction limits, and the workload type. DaemonSet pods or pods with long tolerations may remain for much longer.
So rescheduling should not be treated as proof that the old process has stopped. For stateful workloads, fencing the old node or writer is often just as important as starting the replacement.