I'm in the process of copying around 5 million S3 files from one bucket to another. To start, I need to figure out how many files I really have. However, I'm seeing one number in CloudWatch, but when I run the 'aws s3api list-object-versions' command, it's showing about 10,000 fewer objects than expected. I'm trying to avoid creating an S3 inventory since that takes quite a while, but I'm wondering if that's my only option. Has anyone dealt with this situation before?
3 Answers
Make sure to check if you have versioned files. The API counts the versions too, which could explain the difference you're seeing.
I get that you're not keen on using the inventory feature, but have you tried S3 Storage Lens? It can provide real insights and metadata about your S3 usage without the wait.
Just a thought: the 'list-object-versions' command might be paginated, so there could be more results available on the next page. Did you check if you're reaching the pagination limits?
Yeah, considering I have 5 million objects, there's definitely a lot of pagination. I doubt that's the issue.
Good point! I'm thinking that the S3 API does include those versions in the count.