What happens to an RKE2 cluster if all three embedded etcd servers get new IP addresses? How would you bring the control plane back?
How do you recover RKE2 if all three etcd server IPs change?
Answers
If etcd still has the old peer addresses, the members may be unable to communicate and form quorum. Without a functioning etcd, the Kubernetes API and control plane cannot work normally. Existing workloads may keep running for a while, but scheduling and changes through the API will be affected.
For recovery, use the RKE2 cluster reset procedure rather than trying to edit the Raft membership by hand. Stop the server processes, preserve the available data and snapshots, then reset one server as the sole etcd member. Once it is healthy, bring the other servers back using the RKE2 rejoin procedure. Check the version-specific RKE2 documentation for the exact commands and data directory steps.
Reply to this answer
I'd treat this as a disaster recovery operation, not just an IP edit. Pick one server with usable etcd data, keep the other two stopped so they cannot complicate recovery, and rebuild membership from that server. Take or preserve a snapshot first if you can.
After the control plane is healthy again, check the other servers and workers, then reconcile your configurations and redeploy applications if needed. Also fix the address allocation so a routine network change does not put you back here.
Reply to this answer
There are two separate things to check: etcd peer connectivity and how clients reach the control plane. A reset may recover etcd, but you still need to verify that the API endpoint and server configuration point to reachable addresses after the move.
Reply to this answer
Add your answer
Answers are reviewed before publication.
5c~gasketwhisk_274:
A reset can use existing data or a snapshot, depending on the recovery path. Check which one you are using before proceeding, since restoring an older snapshot can lose newer cluster changes.