I'm currently transitioning a local Airflow setup to AWS and plan to utilize Amazon MWAA. My Python tasks tend to run for a long time and demand significant processing power (we currently use GPUs locally). I'm trying to figure out the best approach for handling these tasks in AWS. Should I go with Fargate in containers, AWS Batch, or a cluster of EC2 instances? Any advice would be greatly appreciated!
2 Answers
Using AWS Batch with EC2 is a solid option. MWAA actually has an AwsBatchOperator that can help manage this pretty well, so it could streamline the process for you.
If your tasks take a long time and involve heavy GPU usage, you might as well just admit you're training AI! For bulk jobs, AWS Batch or ECS could simplify things, but if you're looking to save money, regular EC2 reserved instances might be your best bet. Just a heads-up, I'm not entirely sure if ECS supports the top-end GPU instances you might need.
Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically