Hey everyone! I'm new to AWS and recently learned about Fargate. It seems like a great option since it manages the infrastructure for you instead of using EC2 instances. I'm planning to host around 20-25 large language models (LLMs) for a web app that lets users choose their own personal assistant model. I'm curious if using Fargate is a good idea for this purpose. Plus, I'd like some help understanding how to estimate the costs for this architecture.
When I checked the pricing calculator, I stumbled upon the terms 'pod' and 'tasks,' and I'm not quite clear on what they mean. Specifically, I need clarification on how many tasks I should run and whether I should keep each model in its own task. I'd appreciate any insights or advice you might have!
4 Answers
A task is essentially a set of containers defined when you create a task definition in ECS, while a pod serves a similar purpose in EKS. If you're looking to host different models like GPT-4, Claude, and LLaMA, it's a smart idea to put each model in its own task. Yes, you'd want one model per task for better organization and performance. Here’s a link to the details on that: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html?icmpid=docs_ecs_hp-task-definition.
Just a heads-up, many LLMs require GPUs for optimal performance, and currently, Fargate doesn’t support GPU instances. This could be an issue considering the costs of GPU resources in AWS. Keep that in mind!
Have you thought about using AWS Bedrock? It offers LLMs as a service, which might be more cost-effective and performant than running them in Fargate containers, although you'll be limited to the models they provide.
If GPU support is essential for your models, Fargate isn't the way to go since it doesn't have GPU support yet. As others mentioned, Bedrock might be your best bet.
Thanks for the clarification! I checked out the documentation again and it's starting to make more sense. Just to confirm, having each model in separate tasks will allow me to manage them better, right?