I've run into a real challenge when it comes to recovering a single, important configuration file that was accidentally deleted from a pod's Persistent Volume Claim (PVC). We had volume backups and snapshots available, but given the size of the PVC, restoring the entire volume just to retrieve that one little file felt incredibly inefficient. The typical process I follow involves restoring the full PVC to a new volume, mounting it to a utility pod, and using `kubectl exec` just to find and copy the file. This certainly isn't ideal, especially during an outage! I'm curious about how others handle file recovery from large Kubernetes PVC backups without reverting to a full volume restore. What are some effective workflows or strategies you've established? Is it common practice to mount backups as read-only to a temporary pod for this purpose? Are there any streamlined methods that you've found to work well in a production environment?
5 Answers
I see what you mean about needing the config file on the PVC. If you're able to manage the situation, a simple cron job that archives or copies important files to another PVC could be the way to go. That would give you a quick point of recovery without needing the entire PVC restored every time.
It's crucial to perform backups correctly. Have you tried Velero? They have features for file-level restores; it could really save you a headache in situations like these!
At CloudCasa, we’ve faced similar requests and now provide file-level restores which could solve your issue directly. You should check out their documentation for more on this solution!
Definitely should consider defining that small file as a secret or ConfigMap. Doing so means you won’t lose it since it's backed by etcd. You can mount it directly from the ConfigMap or secret, which may improve your recovery process!
Have you considered whether that critical config file could be managed as a ConfigMap instead? This might help in avoiding the issue of losing it entirely from the PVC. It sounds like it could be a better solution, depending on your context!
Good point, but it seems like the app needs write access to that config file, making a ConfigMap less viable without some refactoring.
Totally agree! Using a ConfigMap would protect the file better. But, if you're stuck with how your legacy app is set up, different PVCs for configs might be a worthwhile thought.