I'm dealing with a challenging situation where my application typically handles around 10-15k requests per minute, with spikes up to 40k. However, today I faced some massive spikes reaching 90k requests per minute on multiple occasions. My setup includes AWS WAF, Application Load Balancer (ALB), and Auto Scaling EC2 instances, which scaled up to 40 servers to manage the load. Unfortunately, this led to a lot of 5XX and 4XX errors during the scaling process. Some requests are less critical and can be processed later. I'm looking for senior-level architecture suggestions to improve how we manage this traffic flow. We're also considering containerization, but our app is closely tied to a local Redis server, and we need each server to have its own Redis instance and PHP horizon. Any advice?
3 Answers
Make sure you're identifying bottlenecks in your system. Consider scaling both horizontally and vertically, and don’t forget about caching on both the server and client sides. You should also optimize your database with indexing and improve how you handle asynchronous tasks to manage the flow better. Efficient database connection pools and optimized queries could make a huge difference too.
It sounds like implementing a message queue could really help you out. I've worked with Kubernetes during high traffic situations, and when facing big fluctuations, we often turn to solutions like RabbitMQ or Kafka. These tools can help you manage request loads even if there's a slight wait time during spikes.
Have you checked how many of those requests are for the same thing? If you have repetitive requests, caching could help immensely. Also, if it's acceptable to queue up some requests, maintaining a dedicated server for handling queues might let you cache certain responses or parts of responses that are frequently reused.
Unfortunately, no requests are identical enough to be cached. Each request creates a unique job that gets pushed into Redis, which is then processed by another layer of servers that aren't registered with the AWS load balancer.