Hey folks! I'm transitioning from a Java environment, where we currently use a Spring Boot batch process to handle millions of users. We're thinking about moving this to Python and I want your suggestions on the best architecture or stack to handle our needs. Here's how our existing system operates: it connects to all major databases and has individual servers that pull in batches of 100 to 1,000 users at a time. We have a thread pool so each user gets processed by its own thread, and once done, messages are pushed to RabbitMQ or Kafka. I'm aware Python has its quirks with CPU-bound multithreading, but I know there are alternatives, like multiprocessing. I'm looking for guidance on the Python ecosystem as we make this switch!
5 Answers
If you're pulling from a database, consider structuring your tasks to assign batches to separate processes. This approach maximizes performance. Also, integrating a master node to fetch data and dispatch it through RabbitMQ could streamline your workflow.
Why switch to Python at all? If your Java system is effectively handling these tasks, it might be worth staying put unless absolutely necessary. Python has its advantages, but expect some sacrifices, especially with concurrency.
Celery with RabbitMQ seems like a solid choice for managing your tasks. It’s widely used and fits well within the Python ecosystem. Just keep in mind that Celery can get a bit clunky for complex workflows, but for straightforward tasks, it should serve you well.
Dramatiq is a great alternative if you want something lightweight and easier to configure than Celery. It’s been effective for my team at scale without the bloat Celery has.
If you're considering the cloud, leveraging serverless functions with something like GCP Cloud Pub/Sub could ease the transition while allowing gradual migration, keeping your current Java processing in the mix initially.
Related Questions
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically
[Centos] Delete All Files And Folders That Contain a String