I'm running a Linux container that continuously fetches data from an upstream system and loads it into a database. We're planning to deploy it on AWS ECS Fargate but need to clarify the resiliency aspect. Since we can't run multiple replicas without causing duplicate entries in the database, we want only one instance to run across multiple zones. If one zone fails, will AWS automatically relocate the container to another available zone? The documentation doesn't seem to address the single instance scenario well. What other solutions can we consider to keep a single instance running while ensuring resiliency in case of zone failures?
4 Answers
Yes, if you configure multiple subnets for your service while maintaining a single instance, Fargate will attempt to deploy a new instance in a different subnet if the current one fails. It might retry in the failed Availability Zone a couple of times before switching. To enhance resilience, you could use a Pub/Sub model with services like SQS/SNS. This way, the upstream system places data in a queue, and you can run multiple replicas to process it without duplication.
Consider using EC2 with an autoscaling group instead of Fargate. Until you can redesign your ETL process to handle idempotency, Fargate may not be the ideal choice in your situation.
Why not stick with ECS altogether? Containers are often easier to manage compared to traditional VMs, and they can speed up your development and deployment processes significantly.
That sounds promising! Using a queue system could help manage the data flow better without risking duplicated entries.