Hey folks, I'm hoping to get some advice here. I'm transitioning from a Java legacy system that uses Spring Boot for batch processing. Currently, this system efficiently manages tasks for millions of users. We're thinking of moving to Python and I want to know what stack or architecture would work best for handling multi-threaded or message queue batch tasks. Here's a quick rundown of how our existing setup operates:
- It connects to a database that supports all the major database types.
- Each batch service runs on its own server and processes a queue of 100 to 1,000 users at a time.
- There's a thread pool in each service, and each user queue item is handled by a separate thread.
- After processing, the service pushes messages to RabbitMQ or Kafka.
I'd love to hear any suggestions for a similar setup in Python. Just to clarify, I've thought this through and have my reasons for switching. I know that while Python can be tricky with CPU-bound multi-threading, options like multiprocessing exist. It won't be straightforward, which is why I'm reaching out for help!
4 Answers
I love Python too, but I’d be cautious about its multithreading capabilities. What exactly are you hoping to achieve by moving away from Java?
What’s prompting the migration to Python? Are you looking to resolve any specific issues? If it's just a language preference, you might want to weigh the pros and cons of multithreading within Python first.
Just curious, why shift to Python for this? Seems like a big change for multi-threading tasks!
If you absolutely need multithreading, consider using languages like C#, Rust, or Go instead of Python. Those have better multithreading support. But if you’re set on Python, then I'd recommend looking into asynchronous queues and using SQLAlchemy for database interactions. Just be sure to stick with core and skip the ORM for performance. For message queues, you can utilize the Kafka package available for Python; it's designed to work well with it. There's a lot you can do without complicating the stack too much!
Related Questions
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically
[Centos] Delete All Files And Folders That Contain a String