I'm assisting a non-profit with their website and recently optimized their virtual machine, bringing their costs down significantly. However, we've now faced a staggering increase in bandwidth charges from $50 to $900 in just two weeks! I attempted to identify the source of this excessive usage using Azure's tools but had no luck. When I looked at the traffic with iptraf, I noticed multiple connections from IP addresses linked to Google services, which seems suspicious. We're considering blocking these requests via robots.txt, but I'm seeking advice on how to pinpoint who's consuming all this data. We only have about 80 GB of content on the server, yet the usage shows terabytes being pulled. Any suggestions would be greatly appreciated!
5 Answers
You should definitely raise a support ticket with Azure. Their billing can be confusing and messy; they might be able to explain the source of your charges. If you're seeing 10 TB of egress, that's pretty wild! They might even adjust the charges once you clarify things with them.
One way to tackle this is to use Cloudflare; it can help mitigate bandwidth costs if you configure your cache settings correctly. Make sure your expires headers are set too. That can really help reduce the load on your server!
Out of curiosity, why are you using a VM to host the website? There might be other, more efficient options out there.
It sounds like your costs are related to data egress from your VM. Remember, the first 100 GB is free, but after that, each additional GB costs, especially if you're pushing crazy amounts like 10.5 TB a month. I'd recommend blocking suspicious IPs and setting alerts in Azure to monitor your usage more closely. Also, consider enabling VNet logging for more insight on traffic, though it might add some costs.
You might want to check out Google's documentation on managing crawl rates, like this [guide](https://developers.google.com/search/docs/crawling-indexing/reduce-crawl-rate). It could give you some ideas on reducing Googlebot's impact on your bandwidth.
Thanks! I'll recommend this approach to the team.
Thanks for the tip, I'll definitely look into configuring Cloudflare!