What’s the quickest way to upload a large number of small files to S3?

0
14
Asked By CloudySky42 On

I'm trying to upload over **53,586 small files** that add up to around **228 MB** to an **S3 bucket**. The upload process is taking forever! From what I gather, the issue is mainly due to the sheer number of files rather than their size. I'm looking for the best methods to speed up this upload process. Any tips?

5 Answers

Answered By QuickUploadQueen On

You should definitely use the CLI with the command `aws s3 sync`. It's way better for handling a large number of files compared to the console. I work with different file sizes and types, and doing it this way has saved me so much time!

DataTransferGuru -

Couldn't agree more! I run my backups this way too, and it's fast enough to make it worth it.

Answered By FileWhisperer88 On

If you're uploading through the AWS console, that's your problem! It can be super slow. I'd recommend using the AWS CLI instead; it speeds things up since it uses multiple streams. Another solid option is using `boto3` with parallel uploads, which can really help too!

DataDynamo23 -

I've had success using `rclone` for data migrations to S3—definitely worth considering!

TechSavvyTom -

The CLI is indeed the way to go! Just remember that it has options for concurrent requests, but those settings are profile-specific, not CLI arguments.

Answered By S3WhizKid On

Check out s5cmd! It's designed for high performance with S3, and it's quite effective for bulk uploads.

Answered By PracticalUploader99 On

It's efficient to say the least! Storing so many small files on S3 can get pricey and slow. If you're not specifically downloading them one by one often, consider zipping them up before the upload. It saves time and helps with costs!

Answered By CompressionFanatic On

If you can, condensing your files into a single archive before uploading would help a lot. S3 isn't really optimized for storing lots of tiny files, and compressing them can improve both speed and cost efficiency.

Related Questions

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.