I'm trying to figure out how to efficiently get customer logs from their S3 bucket into our account for processing. The customer can set up an S3 object creation event notification to send that data to an SQS queue that we manage. However, I'm worried about scalability since the policy that grants them permissions to write to the SQS queue has a size limit. This might mean we'd have to set up a separate SQS per customer or batch, which sounds complicated. I'm looking for a better solution to handle this.
3 Answers
What are you planning to do with these logs? And how many customers are we talking about? If you're just looking to get the logs into your account, S3 bucket replication might be a solution worth considering. Also, it's worth noting that there’s technically no cap on the number of AWS queues, so you could set one up per customer if needed.
You're right, managing an SQS queue for every customer can quickly turn into a logistical nightmare, especially with the policy size limits. A solid approach would be to have your customers set S3 event notifications to send HTTP POSTs to an API endpoint you own, like one through API Gateway. This way, you can direct everything to a single SQS queue or SNS topic for processing. It simplifies things a lot: no limits on resource policies, only one downstream queue, and it scales easily. Alternatively, you could also use EventBridge, but that can get more complex for your customers to set up.
That sounds interesting, but if customers drop their logs into your bucket, wouldn't you face the same policy length problems with needing one bucket per customer?
Have you thought about using SNS? You could have each customer's S3 send events to their own SNS topic, which then triggers your SQS queue. This reduces the need for extensive policy management since you'd only need to handle permissions for the SNS. However, consider how to manage a noisy neighbor situation—if one customer floods the queue with logs, it might delay processing for others. Also, would it be worth exploring Kinesis Data Firehose as an option? Customers could send data directly to an S3 bucket you control, categorized by customer.
That's a good point! I'm not too worried about noisy neighbors since timing isn’t critical for us. But, what about SQS—does it have a limit on how many SNS topics it can subscribe to?
Well, the idea is to process these logs to generate stats on their activity. Right now, we don’t have any customers, but we're in talks with some larger companies. Managing multiple SQS queues wouldn't be fun, especially with high volume.