Is AWS Fargate a Good Choice for Hosting LLMs in My Chatbot App?

0
5
Asked By CuriousCoder22 On

Hey everyone! I'm new to AWS and recently learned about Fargate. It seems like a great option since it manages the infrastructure for you instead of using EC2 instances. I'm planning to host around 20-25 large language models (LLMs) for a web app that lets users choose their own personal assistant model. I'm curious if using Fargate is a good idea for this purpose. Plus, I'd like some help understanding how to estimate the costs for this architecture.

When I checked the pricing calculator, I stumbled upon the terms 'pod' and 'tasks,' and I'm not quite clear on what they mean. Specifically, I need clarification on how many tasks I should run and whether I should keep each model in its own task. I'd appreciate any insights or advice you might have!

4 Answers

Answered By TechieTom119 On

A task is essentially a set of containers defined when you create a task definition in ECS, while a pod serves a similar purpose in EKS. If you're looking to host different models like GPT-4, Claude, and LLaMA, it's a smart idea to put each model in its own task. Yes, you'd want one model per task for better organization and performance. Here’s a link to the details on that: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html?icmpid=docs_ecs_hp-task-definition.

ModelSeeker45 -

Thanks for the clarification! I checked out the documentation again and it's starting to make more sense. Just to confirm, having each model in separate tasks will allow me to manage them better, right?

Answered By SkepticalSara On

Just a heads-up, many LLMs require GPUs for optimal performance, and currently, Fargate doesn’t support GPU instances. This could be an issue considering the costs of GPU resources in AWS. Keep that in mind!

Answered By AlternativeAdvisor321 On

Have you thought about using AWS Bedrock? It offers LLMs as a service, which might be more cost-effective and performant than running them in Fargate containers, although you'll be limited to the models they provide.

Answered By GPUGuru88 On

If GPU support is essential for your models, Fargate isn't the way to go since it doesn't have GPU support yet. As others mentioned, Bedrock might be your best bet.

Related Questions

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.