My daily crawler finds links on my site and checks each unique URL with a GET request. Some external sites block the requests even though the pages open in a browser, so I get false reports of broken links.
Is there a simpler way to tell a removed page from a site blocking an automated check?
Separate the result into working, apparently broken, and unable to verify. A 403 or a site-specific blocking response like 999 doesn't establish that the page is gone. Put those URLs in a review queue instead of calling them dead.
You could try HEAD first and fall back to GET if HEAD isn't supported or gives an inconclusive result. Check your request headers too, including User-Agent and Accept. Even with those changes, some sites will still block automated requests.
Playwright or Selenium can open the links in an actual browser and run the page's JavaScript. That may get you further than a plain HTTP request on some sites, though it won't guarantee access or prove that the intended content is still there.
I'd reserve that more expensive check for URLs the regular checker couldn't verify, rather than launching a browser for every link every day.
For a manageable number of links, keep the crawler just for collecting unique URLs, then open the uncertain ones in browser tabs for a human check. It's not fully automated, but it lets you spot a missing page without treating every bot block as a deletion.
For a service where you need a more definitive check, its API may be worth looking into. That means handling each service separately, though, and it may not cover every kind of link. There isn't a single API in this approach that checks all external sites.
Answers are reviewed before publication.
Related Discussions