What tools or methods do you use for web scraping?

It sounds like you might have been hit by anti-bot protections. When you say it "didn't work," can you clarify what happened? Were you getting error messages or just no data? Knowing your specific goals when scraping can help pinpoint the problem. Definitely be clear about your method and what errors you encountered!

You might want to consider testing your scraper against a duplicate of the page first. It’s also wise to add some delays between requests to avoid triggering any anti-scraping defenses.

I usually use curl for scraping. It's great for fetching raw HTML! When you tried Beautiful Soup, did the requests get blocked after a few attempts? If so, that's probably due to the site detecting unusual activity. I had success with curl when scraping similar sites—no issues with bots!

If you’re working in the NodeJS world, I created a library called Scrapex you might find helpful! I'm using it for my project, and it’s been working well so far.

If you're facing issues with JavaScript-heavy sites, I suggest trying Puppeteer. It can render JavaScript and scrape content effectively since it controls a headless browser.