I'm in Washington, D.C., and my S3 bucket is located in the Ohio region, which was the closest available option. Uploading roughly 500 GB across about 130 files through the AWS web console is taking around two days, even though my internet upload connection is faster than that. Is the browser upload method the bottleneck, and what settings or tools would make this transfer faster?
3 Answers
For a transfer this large, use the AWS CLI or an SDK instead of the browser console. The CLI can split files into parts and upload several parts or files concurrently, which usually makes much better use of your available bandwidth. For example, `aws s3 sync /path/to/local/dir s3://your-bucket-name/destination/` is a good starting point.
Check whether multipart uploads and parallel requests are enabled. The default concurrency may be conservative, so increasing it can help—for example, setting `max_concurrent_requests` to 50 in the AWS CLI configuration. You can also experiment with multipart chunk sizes, but benchmark changes carefully so you don’t overwhelm the connection or computer.
The region probably isn’t the main issue here. From the Washington, D.C. area, a Northern Virginia region may have lower latency than Ohio, but latency alone shouldn’t make 500 GB take two days. Uploading through the website is more likely limiting throughput than the physical distance to the bucket.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures