Why is uploading 500 GB to my S3 bucket taking two days?
Asked by
MellowCedar47
I'm in Washington, D.C., and my S3 bucket is located in the Ohio region, which was the closest available option. Uploading roughly 500 GB across about 130 files through the AWS web console is taking around two days, even though my internet upload connection is faster than that. Is the browser upload method the bottleneck, and what settings or tools would make this transfer faster?
Answers
The region probably isn’t the main issue here. From the Washington, D.C. area, a Northern Virginia region may have lower latency than Ohio, but latency alone shouldn’t make 500 GB take two days. Uploading through the website is more likely limiting throughput than the physical distance to the bucket.
Reply to this answer
Check whether multipart uploads and parallel requests are enabled. The default concurrency may be conservative, so increasing it can help—for example, setting `max_concurrent_requests` to 50 in the AWS CLI configuration. You can also experiment with multipart chunk sizes, but benchmark changes carefully so you don’t overwhelm the connection or computer.
Reply to this answer
For a transfer this large, use the AWS CLI or an SDK instead of the browser console. The CLI can split files into parts and upload several parts or files concurrently, which usually makes much better use of your available bandwidth. For example, `aws s3 sync /path/to/local/dir s3://your-bucket-name/destination/` is a good starting point.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions