I'm new to Kubernetes and trying to understand what happens when a worker node loses network connectivity because of a failure, while the machine and its processes remain running. Do the pods on that node continue working? If Kubernetes later recreates those pods elsewhere and the original node reconnects, how are their data changes handled? Could data be overwritten, duplicated, or appended? I'd also appreciate an explanation of other important failure scenarios involving network partitions in a Kubernetes cluster.
3 Answers
Kubernetes reconciles workload objects and desired pod state; it does not merge or reconcile application data. If a Deployment or StatefulSet creates a replacement pod, the old process might still be writing on the isolated node at the same time. That can result in two active writers unless the storage or application layer prevents it. A standalone pod generally won’t be recreated automatically.
If the node itself is still healthy, its existing pods can usually keep running for a while, but they’re effectively operating blind. They can’t receive new configuration or updates from the control plane, and they may lose access to services, storage, or other nodes. Once the control plane notices that the node has stopped checking in, it can mark the node unreachable and eventually replace managed pods elsewhere. The timing depends on health checks, tolerations, rate limits, and workload type. DaemonSet pods or pods with custom tolerations may behave differently.
A network partition doesn’t automatically stop the processes on the isolated node. The node may not even realize it has been disconnected, so the original pod can continue running while a replacement starts elsewhere.
The result depends entirely on the storage system and database. An emptyDir belongs to the local pod environment, so a replacement gets a new empty directory while the isolated original may continue using its old one until it is cleaned up. A local persistent volume usually keeps the data on that node and normally can’t just follow the pod elsewhere. ReadWriteOnce storage also won’t necessarily prevent the old machine from writing if it is already isolated; the platform may need the node to recover, a timeout to expire, or explicit fencing and force-detach support from the storage driver. Databases must handle replication and conflicts themselves: quorum and leader election can prevent split brain, while multi-writer systems may need conflict-resolution rules. Kubernetes itself won’t overwrite, append, deduplicate, or merge those writes.
For important stateful services, plan for fencing, quorum, and recovery testing rather than assuming that deleting a pod will immediately stop its old process. The storage and database consistency models determine what is safe.
This is why rescheduling should not be confused with fencing. Starting a replacement pod doesn’t prove that the old pod has stopped, so stateful workloads need a way to prevent the old instance from continuing to write.