Hey folks,
I'm currently dealing with a frustrating DDoS attack on my AWS-hosted service and I need some solid advice. Here's the situation:
My setup consists of users accessing our site through AWS WAF, then into an Application Load Balancer (ALB) and finally hitting an EKS cluster where we have both frontend and backend APIs. We're averaging about 20,000 visitors daily, but as a service provider, all our customers are based in the same country.
The problem is that every 10 to 30 minutes, we experience sudden spikes in requests that overload our application. These requests appear valid — they have the right format and show no obvious anomalies. They stem from numerous different IPs, all from within our country, making geo-blocking impossible. Plus, they all use the same legitimate user-agent, so filtering by that is risky since it might block real users.
The only sign I've noticed is a common JA4 fingerprint, but I'm unsure if it's reliable enough to act on. I need help with a few things:
1. How can I block or mitigate this type of attack while keeping my traffic looking legitimate?
2. Is fingerprinting JA3/JA4 a reliable method for blocking bad traffic in production?
3. What AWS solutions do you recommend? I've tried WAF rate limiting, but with the way the attackers rotate their IPs constantly and the sheer volume, a lot still gets through and overloads our APIs.
The specific endpoint facing the most trouble is resource-intensive, so I can't simplify it much. Any advice, patterns, or tools would be amazing.
Thanks a lot!
5 Answers
Sorry that you're experiencing this. I suggest checking out AWS Shield, which specializes in DDoS protection. It might offer you better capabilities against this kind of attack.
Consider making the most expensive features of your service accessible only to logged-in users. This could deter the attackers and help prioritize legitimate traffic. Maybe implement caching for non-logged-in users as well, just to lighten the load.
Have you considered caching responses from those resource-heavy endpoints? Even a short caching time, like 30 seconds, could ease the load significantly on your backend.
Also, just a thought — have you double-checked your frontend for bugs? I’ve seen simple frontend issues create massive spikes like this!
I’ve seen even a one-second cache make a big difference in reducing backend load.
You definitely don't have a one-size-fits-all solution, but I recommend monitoring the IPs hitting your services. Isolate whether they follow a pattern or come from certain ranges. Also, implementing a rate-based layer of rules using your JA4 fingerprint could be beneficial.
Try counting requests based on the JA4 fingerprint. If the spikes match with what you count, that's a good indicator to start blocking.
It might be worth implementing the AWS WAF Challenge. It's known to filter out bots that typically don't handle challenges well, which could help with these invalid spikes.
Shield Advanced could be a strong option, but it does come with a monthly cost, so just be aware.