I'm trying to centralize our logging by sending S3 Server Logs from one AWS account to a different account dedicated to logging for all our organization's AWS accounts. I found that there's no straightforward way to directly send these logs to an S3 bucket in another AWS account even if they're in the same region. To work around this, I'm considering a couple of options: 1) Setting up a weekly DataSync to transfer the logs, and 2) Implementing a lifecycle configuration to delete data older than a week from the original account to minimize costs. I'd love to hear any suggestions or alternative approaches you might have for transferring these S3 Server Logs efficiently and cost-effectively!
4 Answers
Thanks for your input! I've shared this with our S3 team for further consideration. If they find any additional resources to help, we'll get back to you. Meanwhile, check out these official documents on S3 replication and cross-account storage: https://go.aws/3TRiOmm & https://go.aws/3TRRzIk.
You could look into using cross-account bucket replication and set a lifecycle policy for the local bucket to keep data storage minimal.
Your idea of using DataSync along with a lifecycle plan sounds cost-effective and should work well for your needs!
One option is to skip logging to the original account and send the logs directly to your logging account's S3 bucket. This way, you save money on storage. You could also archive the logs to Glacier after a certain period or delete them if they’re no longer needed. If keeping the logs in the original bucket is a must, consider using Lambda to schedule data syncing or use the AWS DataSync service for this.
Unfortunately, skipping the logging in the main account isn't an option because S3 server logging doesn’t support sending logs directly to another AWS account, which complicates things.