Hey everyone! I'm managing a significant amount of data that exceeds 6PB, spread across various locations globally. For example, I have 1-2PB here, 2PB there, etc. I've been trying to use Veeam for cloud backups, but I've hit a wall since it struggles with even 100TB jobs. I'm considering just setting up separate Linux servers to handle rsync jobs from on-prem to the cloud. Has anyone had success with a different tool for managing large-scale backups like this?
5 Answers
I personally back up 11PB with Veeam without much hassle. Just curious, how are you connecting to your remote sites? Do you use WAN connectors for that?
You're dealing with some colossal amounts of data! What’s your daily data change looking like? Also, what kind of internet connection do you have to handle all this?
There’s hundreds of gigs changing daily. Each project file can reach up to half a TB, and I run multiple projects each day. We currently have a 10gb connection, soon to be upgraded to 100gb, with some locations dropping down to 1gb.
For archival storage, the cloud might get pricey. Consider investing in a good tape system instead. If you’re looking for production-level restoration, check out Microsoft's Data Box Heavy, which handles about 1PB. AWS also offers Snowmobile, which is a semi that acts as a data center—ideal for moving large amounts of data!
Are you trying to back up 6PB every day, or is that just the total size of your data? Many cloud providers offer offline sync options. They send you an appliance to ship back after you load it up, then you configure it for deltas with your chosen tool. By the way, have you evaluated whether all of your data needs to be backed up? Cleaning up unneeded data could save a lot of headaches later when restoring!
We’re aiming for an initial backup of 6PB, then syncing deltas after. Most of this data is crucial for revenue, with only about 50GB that might not be worth the effort to clean up.
Before diving into tools, what's your connection speed like? Also, what are your main concerns with backups—like fire, floods, data corruption, or maybe ransomware? These factors play into what solution might work best for you.
In the States, we have 10gb connections that will be upgraded to 100gb soon, and this location holds about 2PB. Other places range from 10gb to residential 1gb connections.
I run multiple jobs, but I'm not sure what's set up with WAN connectors. I’ll check on that Monday.