I recently faced a challenge with my Node.js backend when the volume of my B2B SaaS unexpectedly tripled in just three months. We were using a basic cron-based job system for background tasks like API calls, data aggregation, and report generation, which worked well initially. However, things started to break down when traffic increased. We had overlapping cron jobs leading to resource conflicts, database connections maxing out during busy times, silent job failures that went unnoticed until clients brought them up, and memory leaks from long-running processes. To tackle these issues, I migrated from cron to Bull, a Redis-based queue system, and implemented features like job retries, dead letter queues, monitoring, and read/write database separation. I'm currently trying to figure out the best practices for horizontally scaling workers, deciding whether to switch to RabbitMQ, and managing graceful shutdowns for workers mid-job. I'd love to hear your experiences and recommendations regarding queue systems and any pitfalls you think I should be aware of!
4 Answers
I use Horizon, a Laravel package that manages workers with Redis. It allows for multiple workers across different hosts, auto-scaling, job retries, and failed job monitoring. Just keep in mind, if you rely on Batches, that can create a bottleneck since it uses a relational database instead of Redis for that part. It works great overall!
Switching from cron to Bull/Redis was definitely the right choice. Redis queues handle most SaaS workloads well, but if you need complex job routing, consider RabbitMQ or Kafka. For scaling, just run multiple stateless workers based on your queue depth. Also, make sure to handle SIGTERM gracefully to let workers finish jobs before shutting down. Keep an eye out for long-running jobs that can block the event loop; separating out heavy jobs helps a lot.
Great advice! The SIGTERM handling tip is something I've struggled with, and separating heavy jobs into dedicated workers makes total sense. Thanks for breaking that down!
Honestly, you're posting a lot in a short time, which raises some flags. But just to clarify, I'm not a bot. I run a SaaS platform, and this question about scaling job queues came up after facing serious issues when our order volume tripled last month. If you're interested, I’m genuinely looking to discuss technical challenges.
Congrats on progressing from cron to Bull, that's the right upgrade path for scalability! For horizontal scaling, adding more worker processes is simple with Bull. Ensure your jobs are idempotent to handle possible duplicate processing. For graceful shutdowns, listen for SIGTERM and use `queue.close()` with a timeout to finish current jobs. In most cases, Bull + Redis will serve you very well unless you need complex routing. Just keep an eye on Redis memory and set alerts for dead letter queues to avoid issues! What's your current queue depth like during peak hours?
Thanks for the suggestion! We're on Node.js with BullMQ, but Horizon sounds interesting. I’ll check it out, especially for its monitoring features!