How can I reliably detect broken external links without false positives?
Asked by
MellowCedar42
I run a daily crawler that checks every unique URL linked from selected HTML attributes across my website. The script makes ordinary GET requests, but services such as Substack, SoundCloud, and Facebook often block automated requests even though the same URLs load normally in a browser. This creates false positives when a link is merely protected from bots rather than actually gone. I could integrate each service's API, or use a browser automation tool such as Playwright, Puppeteer, or Selenium, but both approaches have drawbacks. Is there a simpler and more reliable strategy for checking external links while distinguishing genuinely missing pages from sites that block scripted requests?
Answers
A real browser automation tool such as Playwright, Puppeteer, or Selenium is the closest match to your manual test. It can execute JavaScript, follow redirects, and handle the cookies and browser headers that many sites expect. You still won't bypass every CAPTCHA or anti-bot system, though, so browser automation is not a guarantee.
Reply to this answer
Before using a full browser, make sure your basic checker handles redirects and sends sensible headers, including a User-Agent and Accept header. You can try HEAD first and fall back to GET when HEAD is unsupported, but don't treat every 403, 429, or unusual anti-bot response as proof that the link is dead. Mark those URLs as 'unable to verify' and retry them later instead.
Reply to this answer
For heavily protected services, their API is usually the most dependable source of truth, even if maintaining integrations is inconvenient. A practical system can combine approaches: use normal HTTP checks for ordinary sites, use a headless browser for JavaScript-heavy pages, and maintain a small exception list or API check for domains that routinely block automated traffic. This also avoids launching a browser for every single link.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions