I'm curious about two things regarding AI bot defense. First, is it actually worth blocking AI crawlers and scrapers? I have some ideological concerns, mostly viewing AI and its scrapers negatively, but I also recognize that they might have some benefits, similar to traditional search engine crawlers. So, should I consider blocking them or not?
Second, I've set up my personal web server using a Go web app with Caddy as my reverse proxy, but I currently lack any defense against bots. I'm thinking about implementing Cloudflare for protection, but I'm unsure if it's more complicated than a DIY solution. Any thoughts or advice?
5 Answers
You might want to look into using AI tar pits, which are designed to confuse scraping bots by trapping them on pages filled with auto-generated gibberish. This could be a solid way to protect your site from unwanted scrapers! Check this out for more details: [ai tar pits](https://www.pcworld.com/article/2592071/one-rebels-malicious-tar-pit-trap-is-driving-ai-scrapers-insane.html).
Anubis is a great tool for web scraping defense! You can find it [here](https://anubis.techaro.lol/). It could really help you get started with bot protection. It's also pretty popular right now, so definitely worth checking out!
It ultimately depends on what specifically you're trying to protect. Are you concerned about your text or content being scraped, or do you have other sensitive information?
Realistically, you can't completely stop an AI from accessing your site like any regular user. They can still request your pages and parse the HTML, even if you don’t have a public API. However, if you create some hurdles, you'll likely deter most bots even if there’s no unbeatable defense. Remember, it’s all about making it harder, not impossible.
Haha, so that's where that anime-looking girl I've seen everywhere is from! I'll definitely take a look, thanks!