I run a daily crawler that checks every unique URL linked from selected HTML attributes across my website. The script makes ordinary GET requests, but services such as Substack, SoundCloud, and Facebook often block automated requests even though the same URLs load normally in a browser. This creates false positives when a link is merely protected from bots rather than actually gone. I could integrate each service's API, or use a browser automation tool such as Playwright, Puppeteer, or Selenium, but both approaches have drawbacks. Is there a simpler and more reliable strategy for checking external links while distinguishing genuinely missing pages from sites that block scripted requests?
3 Answers
For heavily protected services, their API is usually the most dependable source of truth, even if maintaining integrations is inconvenient. A practical system can combine approaches: use normal HTTP checks for ordinary sites, use a headless browser for JavaScript-heavy pages, and maintain a small exception list or API check for domains that routinely block automated traffic. This also avoids launching a browser for every single link.
Before using a full browser, make sure your basic checker handles redirects and sends sensible headers, including a User-Agent and Accept header. You can try HEAD first and fall back to GET when HEAD is unsupported, but don't treat every 403, 429, or unusual anti-bot response as proof that the link is dead. Mark those URLs as 'unable to verify' and retry them later instead.
A real browser automation tool such as Playwright, Puppeteer, or Selenium is the closest match to your manual test. It can execute JavaScript, follow redirects, and handle the cookies and browser headers that many sites expect. You still won't bypass every CAPTCHA or anti-bot system, though, so browser automation is not a guarantee.
Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically