Hey everyone! I've been working with a legacy Spring Boot batch process in Java that successfully manages millions of users. Now, we're thinking about migrating this system to Python, and I'd love your thoughts on what stack or architecture would be best suited.
Currently, our setup involves a service that connects to a database (we support all major databases) and fetches batches of 100 to 1000 users at a time on separate servers. Each server has a thread pool, processing each user in a dedicated thread, and after processing, we send messages to either RabbitMQ or Kafka.
I'm aware that Python can have issues with CPU-bound threading, but I know there are alternatives like using multiprocessing. I'm looking for solutions within the Python ecosystem that could effectively replace our existing system. Appreciate your inputs!
5 Answers
Dramatiq is another option if you prefer a lighter alternative to Celery. It’s easy to configure and has worked well for my team in a scalable environment.
You might want to look into Temporal. We moved from Celery to it because of its better workflow orchestration capabilities. It’s great if you need to tie multiple tasks together.
I’d recommend using Celery with RabbitMQ. It’s a solid choice that fits well with your requirements for handling task queues efficiently in Python.
If you're set on migrating, consider dividing the workload among separate Python processes ahead of time—it can improve performance significantly. You can also set up a master node to fetch data from the DB and distribute tasks to worker nodes via RMQ.
Honestly, I wouldn't bother switching to Python unless you have a really strong reason. Java handles these scenarios so much better, especially with concurrency and performance.
Related Questions
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically
[Centos] Delete All Files And Folders That Contain a String