I need to transfer roughly 5 TB containing millions of files from an on-premises environment to an S3 bucket. Both `aws s3 cp` and `aws s3 sync` appear to freeze or become unresponsive. I've tried changing some CLI settings without success. Is the AWS CLI suitable for a job with this many files, and what approach or configuration would make the transfer reliable and efficient?
5 Answers
The main problem is probably the file count, not just the total size. Every small file requires filesystem scanning, metadata checks, and network requests, so latency and TCP overhead add up quickly. Tools such as s5cmd or rclone can improve concurrency and often outperform the AWS CLI, but they can’t completely eliminate the cost of millions of individual objects.
A custom batch script can work if it records completed files, retries failures, and resumes safely, but a poorly designed script may be much slower than the CLI. For this amount of data, I’d benchmark s5cmd or rclone and seriously compare them with DataSync before investing time in a bespoke solution. Multipart uploads mainly help with large individual files; they won’t solve the overhead of millions of small ones.
For a production migration of this size, AWS DataSync is probably the best fit. It is designed for moving large datasets with many files, handles retries and progress tracking, and avoids having to maintain a custom transfer script. It may take some setup, but it should be much less fragile than repeatedly running a very large CLI sync.
If you have enough temporary storage, package the data into one or several large archive files first, transfer those, and then extract them on an EC2 instance. A single large transfer can use the network far more efficiently than millions of small uploads. After extraction in AWS, run the S3 upload from EC2, preferably with an S3 VPC endpoint. You can also stream an archive over SSH directly into an extractor if you want to avoid storing the archive, though that is more complex.
The individual-file overhead still exists during the final upload, but doing that from inside AWS is generally much faster and more reliable than performing every small-file operation across the on-premises connection.
You can experiment with increasing the AWS CLI transfer concurrency in its configuration, but tuning threads only goes so far. Splitting the job by filename prefix and running several independent sync processes may help, provided the storage and network can handle it. Don’t blindly start many processes, though—the source filesystem may become the limiting factor, and parallel syncs can make failures harder to manage.
s5cmd is one of the first alternatives I’d test. Also check whether the local filesystem itself is becoming the bottleneck while the source tree is being scanned.