Cloud data transfer charges are making a scraping workload expensive. Would a dedicated server with unmetered bandwidth save enough to justify managing it, and what should I watch out for?
Is moving from cloud hosting to a dedicated server worth it for bandwidth costs?
Asked by
Javrel.2f
Answers
It can make sense when outbound traffic is the main cost. Compare the full monthly bill with a dedicated server, including any limits on the advertised unmetered connection, before assuming the savings will be as large as they look.
The trade is that a managed cloud service takes care of things you may have to handle yourself on a bare metal server. Factor in the time spent keeping it running, not just the hosting price.
Reply to this answer
Don't underestimate the admin work. You may need to set up systemd services for the scrapers, tune the operating system and troubleshoot resource problems yourself. If that sounds like a distraction from the actual work, the cheaper server might not feel so cheap.
Reply to this answer
Sort out remote access before exposing the box. Firewall rules that allow administration only from a fixed IP, or access through a VPN, are options to consider. A tool such as fail2ban can block repeated probes, but it shouldn't be your only layer of protection.
Be careful with custom blocking rules if your scrapers rotate proxies. A rule that catches your own traffic can lock you out.
Reply to this answer
Add your answer
Answers are reviewed before publication.
5h!rivetstamp_380:
Test the provider's remote console before changing access rules. Having a way back in is useful if a firewall or ban rule catches you by mistake.