I'm relatively new to AWS, working as an intern, and I'm developing an internal app for my company. I need to ensure that my Lambda function does not run concurrently since it's modifying a file in S3, which could lead to overwrites. I'm currently using SQS as an intermediary between the trigger and the Lambda function. I'm considering setting reserved concurrency to 1 to limit this, but I'm open to suggestions for better approaches. Any advice would be greatly appreciated!
4 Answers
You could configure the trigger to allow a minimum of 2, while keeping the function's reserved concurrency at 1. This way, the trigger won’t keep trying to invoke more Lambdas. Also, consider using ifMatch in your S3 requests to prevent updates if the object has changed. However, I’d suggest reevaluating your architecture; there might be better ways to manage this.
Quick related question: If I have an SQS trigger set to max concurrency of 2 and my Lambda's reserved concurrency is 1, can the SQS poller face issues due to this limit?
Using an SQS FIFO queue could be your best bet. With a FIFO queue, if you put all your messages into a single message group, Lambda will ensure only one execution runs at a time, which should prevent those overwrites. Plus, FIFO queues handle deduplication, which might be useful in your case.
Definitely set your Reserved Concurrency to 1. This setting defines both the max and min number of executions for your Lambda function, effectively controlling concurrency. It's pretty much the only way to guarantee that only one instance runs at a time.
Yes, in that scenario, the poller will try invoking two Lambdas, but it will get throttled because of your concurrency limit. It's not a disaster, just something to keep in mind.