I'm encountering a frustrating issue with my AWS Lambda function being triggered twice whenever I upload files to my S3 bucket. Here's my setup: I've got an S3 bucket configured to send event notifications to an SQS queue, and that queue is set up as the event source for my Lambda function. I've set the SQS batch size to 10,000 messages and the batch window to 300 seconds, whichever happens first.
For example, when I upload 15 files to the S3 bucket, I'm seeing two Lambda invocations for these 15 messages—one handling 11 messages and another dealing with 4.
What I expected was for a single Lambda invocation to process all 15 messages simultaneously.
I'm wondering:
1. Why is my Lambda being invoked twice despite the generous settings for batch size and window?
2. Is this behavior normal due to the way Lambda and SQS handle scaling and polling?
3. How can I adjust my Lambda or SQS settings to ensure only one invocation occurs per batch, effectively limiting the concurrency to one?
1 Answer
Lambda functions often run multiple pollers, which leads to messages being split among them. With concurrent polling, it’s common to see messages get divided into smaller batches. You can set the maximum concurrency for your event source mapping to 2 to reduce the splitting as much as possible, but getting all messages processed in a single invocation might not be feasible. Try adjusting your Lambda's concurrency to minimize the divide.
Thanks for the clarification! Just to follow up on that: Is there really no way to ensure that all messages (like all 15) are processed in a single Lambda invocation if everything is configured correctly? Also, doesn't this split lead to increased costs due to multiple invocations?