I'm evaluating external orchestration tools for running batch workloads in production at a medium or large organization. Some jobs would run entirely on Kubernetes, while others would be hybrid—for example, starting a legacy workload on a VM and then launching follow-up processing on Kubernetes. What tools are you using, and how have they performed in practice? I'm especially interested in trade-offs, scalability, scheduling behavior, and experiences migrating from older VM-based schedulers.
4 Answers
The best choice depends on the workload. Generic batch processing often fits Argo Workflows, while data pipelines are commonly handled with Airflow or Dagster. Kubernetes autoscaling can also bring worker node pools up from zero when queued batch jobs create demand, which helps keep costs under control.
Argo Workflows is a strong option when the workload lives entirely on Kubernetes, especially if the team is comfortable with Kubernetes concepts. Airflow is another common choice and can launch work as Kubernetes pods, though it feels more Python- and data-pipeline-oriented. Running jobs as pods provides excellent elasticity compared with queue systems such as Celery. For data pipelines, Dagster is also worth considering.
For a hybrid setup, keep the main queue or workflow coordinator above Kubernetes and treat the cluster as one execution backend alongside the VMs. The coordinator can submit a VM task, wait for its completion, and then dispatch Kubernetes work. This avoids forcing Kubernetes primitives to manage VM-specific behavior. Argo is a good Kubernetes backend, but a broader scheduler may be a better top-level controller when legacy systems are involved.
Argo or Airflow will cover the basic orchestration use case, but large-scale deployments often run into scheduling problems rather than workflow-definition problems. Plain Kubernetes Jobs can start part of a distributed workload and leave it waiting for resources. If you need gang scheduling, quotas, or all-or-nothing admission, Kueue or Volcano are worth evaluating. Kueue is relatively lightweight, although its interface is not especially feature-rich.
That makes sense. We still have legacy jobs managed by an older VM scheduler, and the goal is to move only some processing to Kubernetes. I’m trying to understand whether Argo can coordinate both environments or whether the orchestration layer should remain outside Kubernetes.
The main concern with Argo is hybrid scheduling. It works well for Kubernetes-native steps, but it may not be the best fit for coordinating a VM operation followed by Kubernetes jobs.