I'm a developer at Rhesis.ai, where we operate a FastAPI backend alongside a Celery worker to manage our testing of LLM applications. Our system relies heavily on making multiple external API calls and LLM API queries, which is typical of I/O-bound workloads. Currently, we're running into limitations with Celery because it involves a set number of prefork worker processes, meaning we can only handle a limited number of test sets simultaneously, which isn't optimal. While we use asyncio within our tasks to handle multiple requests concurrently, the tasks themselves are executed sequentially at the worker level due to this limitation.
What I'm seeking is a solution that can fully leverage asynchronous programming, allowing tasks to be continuously scheduled onto an event loop without being restricted by the number of worker processes. FastAPI demonstrates the model we're looking for since it effectively scales with multiple processes while handling many concurrent requests. However, FastAPI doesn't include task queuing.
We've also looked at Dramatiq, but it seems to have a similar limitation to Celery. I would love to hear recommendations for a stable and mature library or architecture that can handle this async workflow effectively. We're not interested in adopting an experimental solution for our core infrastructure.
5 Answers
If you're up for a DIY approach, try using ZeroMQ. You can create a lightweight system in about 50 lines of code tailored to your needs. Just make sure to brush up on the right patterns first, because you'll still need to manage how tasks are assigned and their execution timing to avoid blocking issues.
You don't necessarily need to switch away from Celery. A straightforward fix could involve using `gevent` to monkey-patch your solution, allowing you to start workers with concurrency capabilities. A command like `celery worker -P gevent --concurrency=100` should provide the concurrency model you need.
You might want to take a look at TaskIQ; it's designed as an async alternative to Celery, which sounds like it aligns with what you're looking for.
I checked out their docs—looks promising for handling async tasks!
Consider checking out Temporal. It might be a good fit since it allows a worker to leave a task while waiting for I/O to finish, picking up another task in the meantime. Just keep in mind it could be an overkill for simpler workloads, as each request becomes a 'workflow.'
That sounds a bit like what asyncio does, right? Just without having to set up multiple workers?
Oban is another option you might consider. I haven't used the Python version, but the Elixir version was solid. It's backed by PostgreSQL/SQLite, which is great for infrastructure stability and reliability.
Exactly! A custom solution can be quite effective if designed well.