I'm in the process of moving our local Apache Airflow setup to the cloud with Amazon Managed Workflows for Apache Airflow (MWAA). We have some Python tasks that are quite resource-intensive, usually leveraging GPU for processing. I'm trying to figure out the best approach for executing these lengthy tasks in the cloud. Would it be better to run these tasks using Fargate in containers, utilize AWS Batch, or spin up a set of EC2 instances? Any insights would be greatly appreciated!
1 Answer
If you're dealing with long-running Python tasks that require a GPU, you might want to consider AWS Batch or ECS for convenience. They make managing bulk workloads easier. However, if cost is your priority, basic reserved EC2 instances could be the most economical choice. Just a heads up though—I'm not sure if the high GPU instances are supported with ECS right now.
Haha, it's not AI this time, I promise! But can you break down the pros and cons of using Batch versus ECS in this setup?