I'm developing a self-hosted social media analytics tool utilizing SvelteKit, PostgreSQL, and n8n on a VPS. The main feature is to help creators measure their engagement by comparing their performance with similar accounts slightly above their tier. For example, if someone has 2,000 followers, the tool would analyze what 10,000-follower accounts in their niche are doing differently. For my own profiles, I'll rely on official APIs, but I need a scraping solution specifically for public competitor profiles—around 50 to 200 unique accounts that I plan to refresh weekly. While the volume is low, the solution needs to be reliable enough to support a SaaS model.
I've already ruled out some options: Instaloader has been breaking frequently and isn't maintainable at scale; creating something from scratch isn't feasible due to the maintenance burden; and enterprise solutions like Bright Data and Oxylabs are beyond my budget at this early stage.
Currently, I'm looking into Apify actors, as they seem well-established, but their pricing can get tricky depending on usage. I'm also considering ScrapeCreators, though their pay-per-credit model looks promising, I can't find independent validation on their reliability. I'm open to discovering other alternatives as well.
Specifically, I have a few questions: 1. If you're using a similar tool in production, what solution are you utilizing? 2. Have any tools proven stable during Instagram's anti-bot updates in 2024-2025? 3. Are there any cautionary tales or horror stories I should be aware of before choosing a platform?
2 Answers
I’ve been using ScrapingBee for a couple of projects, and it’s been pretty reliable so far. I like that it handles all the rotating proxies for you, which saves a lot of hassle. It’s not perfect, but I've managed to scrape competitor profiles for my analytics tool without running into many issues. Definitely worth considering if you're keeping it low-volume!
Yeah, I’m curious about that too! I want to avoid any nasty surprises.
I’ve had decent luck with Apify actors for similar scraping tasks. They have a massive library of pre-built templates, plus you can build your own if needed. Just watch out for the pricing per usage, which can get weird depending on how many requests you're making. Definitely has its pros and cons, so I'd advise starting with a smaller project to test it out.
I agree! Just make sure to monitor your usage; it can add up quickly.
Good point! I'll keep that in mind when planning my project.
How's the pricing structure? I’ve heard it can vary based on usage.