I'm looking to analyze videos using deep learning models hosted on AWS. The processing for each video will take about 20 to 30 minutes, and I have the models packaged in Docker images with the videos stored in S3. I'm considering using AWS Batch on an EC2 instance to handle long-running workloads that require a GPU. Is AWS Batch a good technical and cost-effective solution for this setup? Also, can I access S3 from the execution environment to load the videos and save the results?
3 Answers
Yes, AWS Batch is a great choice for your needs! Just to clarify, you don't actually 'attach' S3 to Batch. You'll need to set up a job execution role that has permissions to access S3. The container can then download the video files from S3, process them locally, and upload the results back after processing. It aligns perfectly with your goal, and I believe it’s the best approach here.
Thanks for the suggestion! I wasn't aware of them, but I’ll take a look and see if one of those fits my needs better.
Your plan sounds solid! There are also orchestration tools that work well on top of batch processing, which might simplify things for you. For example, Netflix's Metaflow lets you define your workflows using classes, while Dask Coiled sets up its own cluster, allowing for easier management of resources compared to directly using Batch.
Related Questions
Biggest Problem With Suno AI Audio
How to Build a Custom GPT Journalist That Posts Directly to WordPress