I'm working with a standard SQS queue and using a Lambda function that I schedule via EventBridge to run every minute. The goal is to manually receive a set number of messages from the queue, which gives me better control over the processing rate. I'm trying to manage bulk product imports where each message corresponds to an individual product. Multiple users can trigger these imports concurrently, creating a situation where, for example, User 1 might enqueue 100 messages and User 2 might enqueue 10 shortly after. I've assigned the user ID as the messageGroupId.
However, in my tests, it seems that the Lambda function primarily processed all of User 1's messages before even starting on User 2's, leading to a late processing of User 2's messages. I expected SQS to prioritize or interleave messages from User 2 with those from User 1. After reading more about how fair queues are supposed to work, I realized that SQS monitors the distribution of in-flight messages among users and prioritizes those with fewer messages to prevent a "noisy neighbor" scenario. But in my case, since I'm only processing three messages a minute, it doesn't cause a spike. So, I'm wondering if my scheduling approach is interfering with how fair queues operate?
1 Answer
Nope, it doesn't sound like a misunderstanding, you're actually handling deduplication incorrectly. Instead of trying to control the flow this way, consider using the built-in deduplication properties or even a DynamoDB table for managing duplicates. You're slowing things down for a small issue. Just focus on cleaning that up instead!
But my main issue isn't about duplication, it's more about managing the rate of imports since I have to stick to third-party API limits. I just want to spread out the requests better!