What’s the Best Method to Transfer 25-30 TB of Data from HDD to S3?

Uploading such a large amount of data? It’s probably best to skip the network upload entirely. A Snowball device allows you to copy data directly on it and ship it back, which is rather quick—around 1-2 weeks total. Just be aware that costs may vary depending on your needs, and you can find more details on the AWS Snowball console.

While Snowball Edge was highly recommended, it's been discontinued for new customers which complicates things. Instead, you could look into AWS DataSync or transfer family for a smoother experience. What's the timeline for your upload, and what type of data are you dealing with? If it's just over a million images, that might influence the upload strategy further.

If your upload speed is being throttled during business hours, check if there's an egress limit per connection or if it’s just the overall connection speed. Sometimes ISPs have specific restrictions at certain times. You could consider moving the HDD to a different location or renting a server locally to take advantage of higher speeds.

Check out AWS DataSync as a viable option. It can help optimize the transfer based on your available bandwidth and manage retries. I’ve used it in a prior project and it made migrating massive archives a lot easier. Just keep in mind that higher performance options often come with additional costs.

You might want to consider negotiating with your internet service provider or using AWS S3 Snowball for your data transfer. Snowball allows you to physically send data to AWS, which could save you a lot of time if your office bandwidth is too slow for direct uploads.