I've inherited an internal K8s cluster comprised of 5 physical servers, configured as one master and four worker nodes. Unfortunately, it's quite outdated; the servers use either RAID-0 or single disks, and they rely on OpenEBS Hostpath for persistent volumes—this means that crucial applications like a password manager, Nextcloud instance, and a help desk tool run without any backup configuration. The primary goal is to safely transition some persistent volumes (PVs) to NFS for better data security while we reinstall the servers with proper RAID-1 setups.
Given these circumstances, I'm considering using a tool for PV migration and implementing backups via Velero. However, I need advice on the best way to start the server upgrades, especially since the master node currently runs on a single disk. I would prefer a step-by-step approach to avoid downtime and ensure data integrity during this migration process. How should I go about it?
1 Answer
Migrating from hostpath to NFS could slow down your apps, especially if they serve data. I suggest dumping the etcd state first by stopping all worker nodes and the master services. Just make sure to restore it to a fresh, RAID-equipped cluster afterwards. Don't forget to back up the TLS certs too, since you’re running everything in containers. Keep your paths and addresses consistent during the rebuild to avoid sticky issues later on!
You raised an excellent point about the NFS speed. The password manager and other databases might operate slower once migrated, but they're too important to risk losing. Have you tested etcd backups before?