I'm working on several APIs that need to utilize an asynchronous pattern, particularly for tasks that involve database stored procedure calls that can take anywhere from 5 to 20 minutes. I also encounter situations where some jobs require more compute power. I know that using Celery with Redis is quite common for managing async jobs, but I'm curious about how others are implementing it in a production environment, especially when there are multiple APIs with different job requirements.
5 Answers
I've mostly gone with Celery using RabbitMQ as a broker. It’s a reliable way to persist tasks, so if RabbitMQ has issues, you won’t lose your tasks. Although, I have to be honest, I’m not a huge fan of it overall.
Using a file for job status is definitely a clever approach! It may turn into a simple job queue, but it’s probably worth it if you care about reliability, especially for lengthy processes.
In my opinion, that 20-30 minute duration is quite long for asyncio. While it queues jobs and manages other tasks simultaneously, if you get disconnected or something goes wrong, you might lose track of the ongoing jobs. It might be smarter to log the job status to a file and periodically check for its completion.
I’ve switched to TaskIQ with RedisStreams for a modern async solution. However, what you’re describing feels more like durable workflow processes rather than simple tasks. Maybe look into tools like Temporal or Prefect for complex workflows.
Celery and Redis is a solid combo! The central Redis broker allows all your APIs to queue jobs efficiently, plus you can spin up workers for individual services which scales really well in production. Just keep an eye on memory usage in case you end up with huge queues.
Totally agree! It works almost perfectly until you encounter those bizarre silent failures that might go unnoticed for ages. Debugging them later can be a nightmare.