I'm trying to upload over **53,586 small files** that add up to around **228 MB** to an **S3 bucket**. The upload process is taking forever! From what I gather, the issue is mainly due to the sheer number of files rather than their size. I'm looking for the best methods to speed up this upload process. Any tips?
5 Answers
You should definitely use the CLI with the command `aws s3 sync`. It's way better for handling a large number of files compared to the console. I work with different file sizes and types, and doing it this way has saved me so much time!
If you're uploading through the AWS console, that's your problem! It can be super slow. I'd recommend using the AWS CLI instead; it speeds things up since it uses multiple streams. Another solid option is using `boto3` with parallel uploads, which can really help too!
I've had success using `rclone` for data migrations to S3—definitely worth considering!
The CLI is indeed the way to go! Just remember that it has options for concurrent requests, but those settings are profile-specific, not CLI arguments.
Check out s5cmd! It's designed for high performance with S3, and it's quite effective for bulk uploads.
It's efficient to say the least! Storing so many small files on S3 can get pricey and slow. If you're not specifically downloading them one by one often, consider zipping them up before the upload. It saves time and helps with costs!
If you can, condensing your files into a single archive before uploading would help a lot. S3 isn't really optimized for storing lots of tiny files, and compressing them can improve both speed and cost efficiency.
Couldn't agree more! I run my backups this way too, and it's fast enough to make it worth it.