We recently faced a major problem at 3 AM when an integration partner accidentally sent around 100K requests to our API in less than a minute. Our initial setup was the classic API Gateway to Lambda to Database, which worked for a bit until it hit concurrency limits, and the retries started piling up, putting immense pressure on our database. What saved us was implementing a queue in the middle: we restructured to API Gateway -> SQS -> Lambda -> Database. This setup allowed us to buffer requests, control load levels, and monitor our system's performance with CloudWatch. We also introduced a dead-letter queue to handle troublesome messages safely. However, this approach had some trade-offs, like the need for asynchronous workloads and increased complexity. I'm curious to know: how does your team usually manage sudden traffic surges? Do you use autoscaling, queue-based buffering, client-side throttling, or something else?
3 Answers
It's a solid approach you have there! Just a heads up, you should definitely implement rate-limiting for your API Gateway calls on a per-key basis too, just to keep things from getting out of hand. Also, setting maximum concurrency for the lambdas interacting with your database would help control the load.
Have you considered using caching with your API Gateway? That could have mitigated the Lambda concurrency issue. Just curious if you've thought about it, since it wasn't mentioned in your initial setup.
I actually didn't have caching enabled for that endpoint since the requests were mostly unique and user-specific. But you're right, for repetitive calls it can save a lot of costs and help with concurrency. Definitely something to keep in mind!
Two major challenges to watch out for! You've experienced the Thundering Herd problem here. The second, a lesser-known issue, is the amplification attack. Implementing a caching layer could be really beneficial for messages that are frequently requested.
Great points! I did run into the Thundering Herd problem, which is why I capped the reserved concurrency. The amplification attack angle is really interesting; I need to read more about it. Caching responses definitely seems like a smart move to have in place for repeated requests!
Totally agree! I had reserved concurrency on the Lambda that interacts with the database to prevent overwhelming it, and that really helped. I haven't done per-key throttling yet, but it's on my radar now. Thanks for the suggestion!