I'm trying to upload a hefty chunk of data—about 25-30 TB—from a hard drive to Amazon S3. The problem is, the upload speed is being seriously limited by our office's bandwidth. I'm looking for some tips or ideas on how to make this upload process more efficient, particularly for the Frankfurt region. Any guidance would be much appreciated!
6 Answers
AWS DataSync is definitely the way to go. It manages retries and optimizes for your bandwidth, which really helps. I recently moved a whole broadcast company's massive video archive to S3 using it. It was a game changer!
That project sounds fascinating! Can you tell us how you prepared for it and what the execution looked like?
Just a heads up, Snowball Edge isn't available for new customers anymore. It got discontinued a while back, so I wouldn’t rely on that for your transfer.
I appreciate the info! I wasn't aware of that.
Don’t jump to Snowball right away! Depending on your budget, you might find that AWS DataSync, their transfer family, or even multipart uploads suit your needs better. What's your timeline? How old is your HDD and what kind of data are you working with?
I’m concerned about the costs of Snowball. I'll likely go with DataSync since I have over a million images to upload.
With 25-30 TB, I'd suggest avoiding the network upload. Using AWS Snowball Edge would actually be a quick solution where you get a physical device, load your data onto it, and then ship it back to be inserted directly into your S3 bucket. Just a quick note: it's usually a better option than spending weeks on a network upload, and they handle verifying the integrity of the data for you.
This response seems outdated; I thought Snowball was much more expensive now?
Isn’t Snowball Edge discontinued already?
If you haven't already, check out AWS DataSync. It can help optimize your uploads, though just a heads up, faster services usually come with higher costs. Make sure you're not hitting your ISP limits instead of Amazon's. S3 performance can be tied to your payment tier, too.
You might want to consider negotiating with your ISP for better speeds, but a solid alternative is AWS S3 Snowball. It's specifically designed for large data transfers like yours.
Thanks for the suggestion!
That sounds really relevant! Can you share more details about your experience?