I'm working on an app that lets users submit validation tasks, and I'm using Celery with Redis and MySQL to manage these tasks. Each task can take up to an hour to complete, and currently, I'm running Celery in a Docker container hosted by Coolify. My challenge is to find a seamless way to upgrade or redeploy my setup without any downtime or interruptions to running tasks. Coolify supports blue-green deployments, which could be helpful, but my main priority is ensuring that no tasks are lost or halted during the transition. I'm open to any suggestions or experiences you've had that could assist with this issue!
3 Answers
For blue-green deployments, you want to stop new tasks from being accepted on the blue side while the green deployment is running. This is known as a 'soft shutdown'. Once the green is up, send a SIGTERM to the blue deployment, causing the workers to stop accepting new tasks but still finish their existing ones.
From my experience running Celery in production, the best approach is to make your jobs idempotent or incorporate checkpoints for quick resumption. Services shouldn't be treated like cattle; you want to avoid complaints when jobs inevitably fail. It's tough to guarantee zero downtime unless you manage every aspect of the infrastructure yourself, so be careful with your SLAs.
Consider using Testkube for managing your jobs since it's Kubernetes-native. It allows you to run long-lasting jobs as separate pods, so you can redeploy your app without disrupting any ongoing tasks. Jobs finish running in the background and provide results through a UI/API, ensuring zero downtime during upgrades. I've seen many switch from Celery to Testkube for its robust handling of long tasks.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures