So, I have a Lambda function that's set up to process messages from an SQS FIFO queue. The configuration allows for up to ten messages to be processed per batch. Recently, I noticed a strange pattern: from around 6-7 PM for the past two days, the average number of messages processed per batch dropped drastically to about 1.5. This led to an increase in Lambda invocations, as AWS seemed to be firing up more functions to keep up with the flow of messages. By 8 PM, things started going back to normal and by 10 PM, it was back to usual. The weird part is that the overall number of events being pushed to the queue hasn't changed. Does anyone know what might be causing Lambda to suddenly adjust the number of messages it processes in each batch?
1 Answer
This is a really interesting scenario! In general, with normal SQS queues, you have the BatchSize and MaximumBatchingWindow settings that control how messages are collected before Lambda gets invoked. However, with a FIFO queue, it's a bit different. FIFO queues can't use MaximumBatchingWindow, as they have a fixed 0-second waiting period before invoking Lambda. Since FIFO queues can only trigger one Lambda instance per shard per MessageGroupID, it's possible during those peak hours (6-7 PM) your system might be sending more messages across several MessageGroupIDs. This would mean Lambda would invoke instances handling smaller batches more frequently. It might also be worth looking at the median and max batch sizes instead of just averages to get a clearer picture of what’s happening. You might have one group pushing solid throughput while others are sending fewer messages.
Thanks for the insight! I checked our dashboard and saw spikes in inbound messages and increased concurrency, which is nice. However, I noticed the messages per request dropped significantly around 6 PM and stayed around 1.54 with little change. I reviewed how our message group IDs are distributed, and everything looked normal during these times, so I'm puzzled. No single group seemed to have a sudden influx of messages. What do you think could be happening?