Hey all, I'm facing some unexpectedly high S3 costs for a bucket connected to my AWS Storage Gateway. Right now, the bucket holds about 3.6 TB of data all stored in the Infrequent Access (IA) class, but the costs have skyrocketed. After enabling S3 access logging, I discovered an overwhelming number of HEAD and GET requests hitting the bucket consistently. Given that the IA class charges significantly for requests, it's really straining my budget. The cache size on the Storage Gateway is only 80 GB, which doesn't seem to be enough for effective caching, leading to frequent hits on S3. I'm considering a few options, like whether to switch back to the Standard storage class to cut down on request costs despite higher storage costs, or if I should dive into the application side to see if there's a mounted volume causing all these requests. I initially suspected an antivirus on my EC2 instance, which I disabled, but the costs are still high and the requests persist.
1 Answer
I've faced a similar situation before, and it definitely sounds like your Storage Gateway is flooding your S3 bucket with those frequent HEAD and GET requests. While IA is designed to save on storage, it really dings you when requests are high. Given the small cache size, the gateway's constantly reaching out to S3, racking up those request charges. You might want to look at anything mounted on the gateway—file browsers, backup agents, or anything that scans directories could be a culprit. Try unmounting the gateway for a bit to see if it stops the requests. If you can't reduce access frequency, shifting back to the Standard storage class might actually lower your overall costs since it doesn't charge for requests like IA does. Also, consider S3 Intelligent-Tiering as an alternative if you want a balance between cost and request frequency.
I agree with most of what you're saying! But I suggest holding off on Intelligent-Tiering until you pinpoint the source of the requests. Don’t want to complicate things without knowing for sure!