I'm currently managing a bare metal cluster that has Rook/Ceph installed for providing storage solutions — RBD for block storage and CephFS for file storage. I'm using Velero to handle backups to Wasabi, which is S3 compatible. Right now, I've got Kopia set up for data movement, and it works great for RBD volumes. The process involves taking a CSI VolumeSnapshot, creating a new PV from that snapshot, and then using Kopia to upload the data to Wasabi.
However, I've hit a snag with CephFS. Taking a VolumeSnapshot is slow and doesn't make sense for RWX volumes. The snapshots created actually take up the same space as the original volume, which is frustrating because they aren't appearing as CSI snapshots — they exist within the volume itself. This means if I perform daily backups and keep them for 30 days, my CephFS can end up using 30 times more storage than the actual data, even if no files have changed!
I noticed there's an option to set `--snapshot-volumes=false`, but I'm unsure how to implement this specifically for CephFS in a VolumeSnapshotClass. Any suggestions on how to disable snapshots for CephFS while still being able to use Velero?
1 Answer
I faced a similar issue with CephFS where the snapshots were just ballooning my PVC sizes. The fix I found was to create separate Velero backup configurations: one for RBD PVCs using the CSI volume snapshotter and another that directly targets CephFS PVCs with the "File System Backup" method, which utilizes the Velero node agent to pull CephFS data from its mount point on the host.
That sounds like a solid plan. Do you have any configuration snippets handy? Do you match these volumes by labels, or does it handle StorageClasses on its own?