I'm moving around 8 C# applications from Windows Task Scheduler to AWS, and these apps mainly interact with the same SQL Server database to perform some business logic and update data. I have a few questions about the best practices for this migration:
1. Should each scheduled task handle everything from start to finish, or is it better to break things up? For example, should one ECS task handle fetching work items and queue them, while separate tasks are responsible for processing them?
2. Is it better to have one repository for each job or to combine them into a single monorepo?
3. When it comes to managing these jobs, do most people just use CloudWatch and the ECS console, or do they favor a third-party tool (ideally open-source)?
4. What's the standard approach for handling retries? Is it just using CloudWatch alarms with SNS?
3 Answers
I actually went through a similar migration recently and opted to use scheduled jobs in ECS running on Fargate. The key reason we didn’t use Lambda was due to the 15-minute timeout. For our code structure, we went with a monorepo to share the database logic, which has been working well for our team.
If ECS is a must, make sure to leverage the capabilities offered. For task management, it’s generally better not to overcomplicate things. Running a task that handles everything could be more efficient rather than breaking it up unnecessarily since ECS tasks can run indefinitely. Regarding repos, keep it aligned with your organization's approach. Monorepos can work, but if it doesn't fit your workflow, go with what's already in place. Many teams have transitioned to using Terraform for job management instead of relying solely on the ECS console. For retries, consider using an event to SNS, then to SQS, incorporating a dead letter queue to catch any errors. You can monitor dead letter queues through alarms.
If any of your jobs are under 15 minutes, definitely look into AWS Lambda with EventBridge. It can really simplify things and often comes out cheaper. But remember, if some jobs exceed that 15-minute mark, you might still need ECS. As for your repository structure, if half can be transitioned to Lambda and the others can't, maybe consider a mixed approach: keep the ones that need to stay on ECS separate or in a monorepo depending on your team's preference.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures