I'm working on a project that requires parsing data from several websites, but I'm facing some challenges. Initially, I tried using HTTP requests in Python with aiohttp, but since these sites don't have public APIs and are dynamic (the content loads via JavaScript), I wasn't able to retrieve the data effectively. I then switched to using Playwright in Python, which does work, but comes with its own set of problems: it consumes a lot of system resources, it's slow due to waiting on pages to load, and opening thousands of tabs doesn't help either. I've heard there are AI parsers available for scraping, and I'm curious if Playwright in JavaScript could be faster, but I'm not sure. So, I'm looking for advice: is there a more efficient way to grab data from these kinds of websites or ways to optimize my current approach?
1 Answer
You might want to try mimicking the requests that the actual website's JavaScript makes to retrieve dynamic content. Use the Network tab in your dev tools to see those requests and see if you can capture the data that way without scraping the entire page. It could save you a lot of resources and time!
That sounds interesting! I heard that with Playwright, you can intercept those requests directly, which might help you fetch just the data you need without rendering the whole page. Have you looked into that yet?