I'm aiming for zero operational costs while scaling my project, so I bypassed setting up a traditional backend server entirely. Instead, I've been utilizing scheduled GitHub Actions that run twice a day. These actions trigger Supabase Edge Functions which execute scraping scripts using Playwright and Cheerio, verifying pricing data and writing it directly to a Postgres database. While everything is working smoothly now, I'm concerned about the scalability of this setup and potential rate limits on GitHub Actions as my data grows. Has anyone else used GitHub Actions heavily for cron jobs? What are the risks I'm overlooking by not using a dedicated server?
1 Answer
Using GitHub Actions for scheduling is definitely a smart way to manage costs while scaling up. Just a heads up, you might want to run your scraping code through a test platform like Runnable first. This can help you catch any potential rate limits before they happen with the Actions.
When you say Runnable, do you mean using the actual platform or just testing it locally to hit limits? Either way, I'll definitely mock up a pipeline before going full throttle with heavy scraping on GitHub Actions.