I've just started at a new company and I'm analyzing our S3 expenses. We're using Intelligent-Tiering on a bucket containing 66 million small objects, each about 300KB, totaling around 19 TB. However, our monthly bill is around $2,000, which seems much higher than expected since my calculations for storage and monitoring suggest it should only be about $400-500. Interestingly, standard storage pricing would actually be more costly than Intelligent-Tiering for this amount of data, which leaves me puzzled about the additional $1,500+.
I'm curious if Intelligent-Tiering tends to get pricey when dealing with such a large number of objects. What should I look out for in Cost Explorer — like requests, transitions, retrievals, or inventory? Also, have any of you experienced high costs from large buckets and shifted away from Intelligent-Tiering because of unexpected charges? I'm also interested in any tools or dashboards that can help break down S3 usage when dealing with a substantial bucket. Currently, the storage and monitoring costs look normal, so something else must be inflating the bill. I'd appreciate insights from those who've tackled similar issues with millions of small objects in S3.
5 Answers
Definitely do a deep dive into what’s actually causing those charges. The monitoring fee should be about $150 for that number of objects, so it’s likely the requests causing the spike. Also, if you're using bucket encryption with KMS without a key, frequent access could shoot your costs up quickly. Intelligent-Tiering might not be the biggest culprit here, it’s worth examining the requests more closely.
It sounds like you might want to take a closer look at your request charges. If you're frequently accessing or transferring objects to another region, that can drive up costs significantly. Keep in mind that Intelligent-Tiering might not provide immediate savings in the initial months, so factor that in.
According to the pricing details from AWS, there's a monitoring fee for Intelligent-Tiering of about $0.0025 per 1,000 objects each month, which adds up to around $165 for your 66 million objects. The bulk of the unexpected charges likely comes from the requests. For instance, LIST/HEAD/PUT requests are $0.005 per 1,000, and lifecycle transitions are $0.01 per 1,000 objects. Just one lifecycle transition could rack up costs of hundreds or even thousands. You can check out the pricing details on AWS's site for further clarification.
When considering switching storage classes, keep in mind how monitoring charges work. If Intelligent-Tiering monitors an object and changes its class, it may check again later. It can potentially lead to repeated costs if the object's status changes too frequently.
You should also look into your overall data access patterns. Even if you're monitoring correctly, unexpected access can lead to high costs if you're not careful with how many requests you make or how often you transition between storage classes.
Yeah, it actually takes 30 days of no access for the objects to transition from frequent to infrequent access. That could be affecting your costs too.