I'm trying to block AS16509 due to it only having bot traffic and not being listed in any managed block lists. The challenge is that the IPs from the crawler are very dynamic, which means I need to block the entire ASN. I've downloaded all the CIDR ranges and compressed them, but it ends up being over 3000 ranges. While creating the IPset with Terraform is quick, adding it to a WebACL rule in my WAF makes the apply process incredibly slow—taking about an hour. Is this an issue with the AWS Terraform provider, or are there any alternative methods I could use?
3 Answers
Have you tried putting CloudFront in front and then blocking by ASN? They offer a custom field that logs this kind of data. Check out the [CloudFront headers documentation](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/adding-cloudfront-headers.html#cloudfront-headers-viewer-location) for more info.
Just a heads up, AS16509 is actually Amazon's ASN. Are you sure you're not blocking yourself inadvertently?
It's one of their ASNs, but I host in a different region, so I think I'm safe. [Here’s a link](https://technerd.pro/asn-networks-you-should-block-to-stop-bad-bots/) I found that shows networks to consider blocking.
I suggest enabling debug logging to see what responses you're getting when you call the AWS APIs. It might reveal an error that's causing repeated retries, making the process slower.
I've tried it with smaller IP sets. A set of size 500 takes about 10 minutes, which is manageable, but once I go bigger, it slows way down. Even combining smaller sets leads to issues when applied to a WAF Rule.
That sounds interesting! But I was under the impression that for DDoS protection, it's better to have the WAF in front of CloudFront. I don’t want to incur additional costs from blocked requests if I do this.