How can I prevent Celery tasks from being lost during ECS deployments and scaling?
Asked by
MellowHarbor27
I'm running Celery workers on AWS ECS and want to make sure queued work is completed reliably during production deployments, autoscaling, and task shutdowns. What Celery settings and task-design patterns should I use to handle worker crashes, forced termination, and message-delivery failures?
I'm considering late acknowledgments, requeueing tasks when a worker is lost, limiting prefetching, retrying broker connections and failed tasks, and confirming message publication. I'm also interested in whether batching or a scheduler-plus-worker fan-out design is safer than putting a large amount of work into one task. Similar concerns apply to services with short shutdown grace periods, such as Heroku or Azure App Containers.
Answers
Make every task idempotent. With late acknowledgments, a task can run again if the worker dies after performing the side effect but before acknowledging the message. The task should therefore safely tolerate duplicates, usually by using an idempotency key, a database constraint, or a check that records completed work. Also configure the ECS `stopTimeout` longer than the expected task duration when possible, so workers have time to shut down gracefully instead of being killed with SIGKILL.
Reply to this answer
Use at-least-once delivery as the model, not exactly-once execution. Set `task_acks_late=True` so a task is acknowledged after it finishes, and consider `task_reject_on_worker_lost=True` so tasks are requeued when a worker disappears. `worker_prefetch_multiplier=1` prevents a worker from reserving a large batch that could become delayed or get lost during a crash. Also enable broker startup retries, publisher confirmation where supported, and exponential task retries.
MellowHarbor27: That matches my understanding: these settings improve recovery, but they don’t provide exactly-once processing.
Reply to this answer
For large workloads, split the work into bounded units. Batch processing can handle a fixed number of records and enqueue the next batch, while a fan-out design can use one scheduler task to discover work and separate tasks to process each item. Smaller tasks are easier to retry and distribute, although you’ll need to account for partial completion and duplicate submissions. It’s also worth adding your own dead-letter or failed-task storage because Celery and the broker won’t automatically give you a complete replay workflow.
Reply to this answer
Another option is a managed job service that delivers each job to an HTTP handler and redelivers it when the handler doesn’t report success. That removes some broker, visibility-timeout, and worker-shutdown concerns, but it requires changing Celery workers into HTTP handlers and won’t automatically replace Celery workflows such as chains, groups, or chords.
Reply to this answer
Add your answer
Answers are reviewed before publication.
LunarCedar6: Exactly—Celery gives you at-least-once delivery, not exactly-once execution. Duplicate-safe task logic is essential.