I'm building a system primarily focused on handling network and I/O-bound tasks with specific needs. Most tasks run for less than 10 minutes, but some can take several hours and can't be split into smaller parts. Additionally, the system needs to handle varying workloads efficiently. My current plan is to use AWS Lambda for the short tasks and AWS Fargate via ECS for the longer ones, with messages routed through Amazon SNS to trigger these functions. However, I'm seeking advice on optimizing for cost efficiency and whether this design is solid overall. Any insights would be greatly appreciated!
2 Answers
You're on the right path with using Lambda for short-lived tasks and Fargate for longer ones. This combo is great for mixed workloads! Here are a few tweaks I suggest:
1. Switch from SNS to SQS for handling messages. SQS has better mechanisms for retries and error handling, which is crucial for unpredictable tasks.
2. Consider using Step Functions to manage your workflows. They can help coordinate the execution and retries of tasks, whether they're short or long. Plus, you can visually track everything.
3. For Fargate tasks, utilize Step Functions’ callback pattern to pause workflows until your tasks finish instead of polling repeatedly, which saves resources.
4. If you're worried about costs, Fargate Spot is a cheaper option for non-urgent tasks.
5. Don’t forget to monitor performance with CloudWatch to adjust your resource allocations accordingly!
Why not just run everything on Fargate? Since you're already setting it up for the long tasks, adding Lambda can complicate things unnecessarily. If your tasks don't require a lot of resources, Fargate might handle everything well enough.
I'm concerned about costs. Lambda could be more cost-effective for short tasks that don't need as many resources.