I've recently started using LLM-based scraping, but I'm really struggling with bot detection. It feels like I'm constantly fighting against captchas and getting IP bans. I've tried rotating IPs, but it still seems like a risky game. How do you manage to scrape effectively without getting caught? I'd love to hear any tips or tools that you find helpful!
3 Answers
Sometimes, it's best to politely back off. If they really don't want scrapers, they usually have solid protections in place. Just find a different target!
If you're facing tough bot detection, it might be a hint to reconsider your approach. Many sites have implemented measures to keep scrapers out, and it's not exactly a friendly environment. Maybe think about exploring other career paths or ways to engage with data ethically?
Honestly, I avoid scraping websites that make it tough for me to do so, especially at scale. It's just not worth the hassle!
It's not like scraping is going to disappear; there's always needs for it! Just adapt and keep finding ways to work around the barriers.