Hey everyone! I'm fairly new to AWS and currently working on an internal app as part of my internship. I've come across a challenge where I need to ensure that my AWS Lambda function doesn't run concurrently since it's modifying a file in S3. If the function runs at the same time, changes could get overwritten, which would be a problem. I'm using SQS to trigger my Lambda function, and I'm considering limiting its reserved concurrency to 1. Is this the best way to handle this situation, or are there better solutions out there? I really appreciate any insights you can share!
4 Answers
Quick follow-up! If I set the SQS event source max concurrency to 2 and my Lambda's reserved concurrency to 1, will the SQS poller error out because the Lambda's concurrency is lower than the event source's limit?
Setting Reserved Concurrency to 1 is indeed a good call! This setting limits both the max and min number of concurrent executions for your function, ensuring that only one instance runs at any time. It's definitely the best solution here.
You might want to consider using an SQS FIFO queue for this. With a FIFO queue, the concurrency is controlled by the number of message groups. If you keep all your messages in a single group, Lambda will only execute one at a time. Plus, FIFO queues handle deduplication, which could be useful for your scenario.
I already have a FIFO queue set up with everything in one message group, but I’m still seeing two concurrent executions sometimes. Any thoughts on that?
You could set scaling on the SQS trigger to a minimum of 2 and still reserve concurrency at 1. However, the first option stops further Lambda invocations, while the second just ends up throttling the excess calls. Also, using ifMatch for S3 requests can help prevent overwriting the object if it has changed. Just a thought, though—maybe rethink your architecture to see if there’s a better way to handle the requirement.
Yes, the poller will attempt to invoke with 2 concurrent executions and end up being throttled. It's not a major issue, though!