I'm running backup jobs that transfer large datasets between various systems and S3. To save on costs, I've archived older data to Glacier. However, sometimes these jobs need to access old files stored in Glacier, which leads to failed requests because the objects require a manual restoration. I'm wondering if there's a way to set up an automatic restoration for any files that fail to be accessed in [bucketname]? Ideally, I want the file to be restored for a period, say 7 days, to allow my job to successfully run. Any advice would be greatly appreciated!
3 Answers
Currently, S3 doesn't provide a direct way to trigger actions based on GetObject events. One workaround is to enable CloudTrail Data Events for your S3 bucket. This will log all access attempts, and you can set up an EventBridge to use those logs as triggers. Just keep in mind that this can generate a lot of data, so filtering and managing that data with lifecycle rules is essential. Also, check out this blog that details similar setups for notifications related to object downloads!
I think it's important to note that while that can work, you have to watch out for costs; CloudTrail can rack up expenses quickly, so be discerning about which events you track.
You might want to catch errors client-side if you're working with logs during your job runs. That way, you could manage failed requests more effectively without solely relying on AWS services.
As previously mentioned, CloudTrail is an option, but it’s pretty pricey. Instead, you could regularly check your job history logs for failures and batch restore the files when needed. This may be more cost-effective in the long run. You could also consider adding some monitoring for how often these access issues pop up, helping you to optimize for costs later.
Thanks for the tips! I’ll definitely explore the CloudTrail option.