I help a nonprofit organization preserve film, paper, research, and finished-art archives. We currently have the only copy of roughly 40 TB spread across three heavily used hard drives: two 10 TB drives and one 24 TB drive. The drives are nearly full, and individual files range from tiny documents to files around 800 GB.
We recently received 100 TB of cloud storage through a nonprofit Google Workspace plan, and for now it is our only realistic backup destination. We would like to buy new drives and create another physical copy later, but our budget is limited at the moment.
We are inexperienced with transfers of this size. Tools such as rclone seem promising, but I am not yet comfortable configuring them safely. There also appears to be a daily upload limit of about 750 GB per user, which could make transferring 40 TB take a long time.
What would be a sensible, safe workflow for transferring this data? I especially want to avoid accidentally deleting or corrupting the original files, and I would appreciate advice on verification, upload planning, and how to handle the older source drives.
3 Answers
A low-cost cloud destination can solve the immediate storage problem, but it does not protect against accidental deletion, account problems, or a later cloud-side failure. Keep versioning or retention features enabled if available, restrict who can delete data, and document the folder structure and account access.
When money becomes available, create a second physical copy on new drives and store it separately. For irreplaceable archives, the long-term goal should be at least two verified copies, ideally in different locations, with periodic checks that the files still match their recorded hashes.
Because these drives contain the only copy, treat the first transfer as a copy rather than a move. Do not use a command that deletes source files or mirrors deletions until you have an independently verified second copy.
Copy one drive at a time, keep the originals untouched, and use a tool that can resume interrupted transfers. Start with a small test folder and use a dry-run mode before committing to the full job. After each batch is uploaded, compare file hashes or use the transfer tool’s verification option. Keep a written log of what was copied and verified.
The source drives are also at extra risk because they are old and nearly full. Reading every file may expose a failing drive, so avoid unnecessary scans and make another physical copy as soon as funding allows. Cloud storage is useful here, but it should not be considered the final backup strategy by itself.
The upload quota is the main scheduling constraint. At 750 GB per user per day, 40 TB would take roughly 54 days for one account under ideal conditions, and real-world interruptions could make it longer. If the service terms allow it, multiple properly managed accounts or a shared storage area may increase the available throughput, but confirm the limits and permissions before designing the workflow around that.
Your files under 5 TB should fit within the single-file limit mentioned for the service, so files around 800 GB are not inherently a problem. However, check how large files behave with copying and organization inside the cloud system before starting the full archive. Uploading directly from the original drives is safer than uploading and then trying to rearrange huge files afterward.
This is currently the only copy, so I agree that nothing should be moved or deleted during the upload. I was considering rclone, but I will test the configuration on a small folder first and make sure verification is enabled.