I'm a developer who has taken on the CTO/sysadmin role at my company, where we manage our own bare metal servers in France. Recently, we faced a significant issue where botnet traffic was overwhelming one of our clients' websites, causing it to intermittently go down. This was triggered by URLs linked to marketing campaigns that were being shared broadly. Initially, I thought it was just a spike from a newsletter the client sent out. However, when other sites started dropping as well, it became clear something was wrong. I found that the traffic was coming from many unusual IPs, including those from data centers and VPNs around the world, which suggested it wasn't legitimate. To mitigate the problem, I blocked entire ASN IP ranges using HAProxy tied to specific domain and campaign terms, resulting in a quick decrease in traffic. While this worked, I'm concerned about the root cause, particularly that the database connections were maxed out. I'd like your thoughts on how I can improve this setup—should I stick with this ASN blocking method or explore other solutions?
3 Answers
There’s a lot to consider here. First off, you might want to treat this like a DDoS situation. A CDN or WAF can help mitigate that kind of traffic. Beyond that, think about load testing your web server to understand its limits. Implementing database connection pooling, caching, and maybe even read replicas could alleviate some pressure. Remember, sometimes unexpected traffic surges happen, like when clients send out marketing emails. Make sure to check if that’s what triggered the spike because spam checkers can also generate a lot of unwanted traffic!
Using a service like Cloudflare could be a game-changer. It allows you to manage traffic and set rules instead of just blackholing ASNs. It might also provide better protection against DDoS attacks if that’s what you’re facing.
Your current fix seems like a decent temporary measure, but the real issue appears to be the database exhaustion. It's crucial to focus on optimizing your database next to prevent this from happening again.
I hear you, but I worry about Cloudflare going down—then everything goes down with it. Still thinking it over.