I run a small bare-metal cluster for academic projects and a self-hosted file service. Recently, automated scripts began sending large numbers of requests through rotating residential proxies and scraping random paths. The requests are mostly useless 404s, but they still reach the ingress, consume CPU, and cause the Horizontal Pod Autoscaler to add replicas. Nginx rate limiting has not helped because the traffic comes from many different addresses. I am considering moving verification or filtering outside the cluster, possibly using a proof-of-human or proof-of-work gateway. What are practical, affordable ways to keep this background internet traffic from overwhelming the nodes without buying an enterprise WAF?
4 Answers
The main fix is to filter traffic before it reaches Kubernetes. Put a CDN, reverse proxy, or challenge gateway in front of the ingress and enable bot protection or reputation-based rules there. A self-hosted option such as Anubis can provide a challenge page without requiring an enterprise WAF. Also cap HPA's maximum replicas immediately so a burst of junk traffic cannot consume all your power and capacity.
CrowdSec or Fail2ban can help when you have useful behavioral signals, but they are less effective against large rotating residential networks and may need to run at the host or edge layer rather than inside the cluster. IP reputation rules, connection limits, and a reverse proxy can reduce the noise, while exposing services on unusual ports may deter simple scanners but will not stop determined scrapers.
Fail2ban is awkward to integrate natively with Kubernetes, so treating it as a host-level or gateway control is usually simpler. It should complement, not replace, an edge challenge and conservative autoscaling limits.
Set a sensible maxReplicas value and change the scaling signal to represent useful work rather than total traffic. Queue depth, authenticated jobs, or application-specific work are better metrics than raw requests or generic CPU when the endpoint is exposed to the public internet. Edge filtering should still come first, but this prevents bots from turning directly into extra pods.
Do not scale the real application based on raw request rate or CPU if unauthenticated traffic can inflate those metrics. Put suspicious requests on a separate, deliberately limited path or service, and consider a tarpit for repeated invalid URLs. You can detect patterns such as excessive 404s and route those clients to a non-scaling backend instead of letting them trigger normal replicas.
Dropping unknown routes at the ingress is still useful, but it will not help when valid-looking paths are being scraped. In that case, detection and routing based on behavior works better than a simple deny list.
If you use a third-party edge provider, the free or low-cost tiers are often enough for basic proxying and bot rules, but a self-hosted challenge service is a better fit if privacy and control matter more.