I'm assisting a non-profit with their website setup and after optimizing their virtual machine costs down from $250 a month to less than $100, they've suddenly received a bill for $900 in bandwidth usage within just two weeks. I looked into Azure's web tools to identify bandwidth hogs, but didn't have much success. Instead, I used iptraf and noticed numerous open IP addresses, roughly around 142.251.34.202, which seem linked to Google's services. It appears Google is accessing the website frequently from multiple IPs. I'm not entirely sure if this is the root of the bandwidth issue, but it looks suspicious based on my findings. We're planning to use robots.txt to block unwanted access, but I'd love to hear any simple methods to determine who is pulling so much data. The server has only 80 GB of data on it, but some devices are draining terabytes of data. Any help would be appreciated!
5 Answers
You can check this guide on reducing crawl rates from Google. It might help manage how often they access your site, which could reduce the bandwidth.
I’d recommend opening a support ticket with Azure. They can provide clarity on your billing and may be able to help identify the source of such high egress traffic, which seems unusually high. There's a good chance they could rectify the billing issue if they find an error in their logs.
It sounds like you're dealing with some hefty egress traffic costs associated with your VM. The first 100 GB is free, but after that, it can quickly add up, especially if it looks like you're transferring around 10.5 TB in a month! I suggest blocking those suspicious IP addresses with a firewall and setting up alerts for cost monitoring in Azure. It could also help to enable VNet logs and traffic analytics for better insights, though keep in mind that those features may incur extra costs.
You might want to consider using Cloudflare. If your site's expiration headers are set correctly, it can significantly reduce bandwidth costs, potentially eliminating them altogether.
Thanks for the suggestion!
Just curious, why are you using a VM for hosting your website? There might be more efficient solutions depending on your needs.
Thanks! I'll make sure to recommend it.