How Can I Scale Web Scraping for 500k Hotel Reviews Without Getting Blocked?

Data center IPs usually face blocks. You could try proxy services or something like firecrawl for better results. This could help mask your requests.

Consider running Tailscale on your home network and your EC2 instance. That way, you can use your home network as an exit node, making it appear that the scraping is coming from your home IP. It's cost-effective, and while home IPs may have some rate limits, they generally aren't blocked outright because they get reassigned by ISPs frequently.

From my experience, the issue with being blocked on EC2 is mainly due to the IP reputation. Home IPs are generally clean and won't get flagged as quickly, whereas AWS IPs often get abused by scrapers and are blocked right away. A few options to consider on EC2:

- Rotate through multiple EC2 instances across different regions or use spot instances to change your IP frequently.
- Incorporate random delays of 10-30 seconds between requests, along with realistic browser headers and slow scrolling actions.
- Leverage tools like Selenium with undetected-chromedriver or Playwright to mimic real user behavior.
- Implement batch scraping, targeting 5,000-10,000 hotels per day, instead of overwhelming the server.

This slower approach can work well since you’re okay with not scraping everything at once; many do large hotel scraping projects like this without expensive proxies. Have you implemented random user-agents or delays yet?

If you're open to using proxies, you might want to look into dedicated services like Bright Data or similar providers to help handle the scaling.