I'm trying to figure out how to smoothly handle database migrations in my Kubernetes cluster for Symfony and Laravel applications. I use Kustomize, and my applications are managed as part of an ApplicationSet with ArgoCD.
I've run into some issues:
1. **Init containers**: They can start multiple times during scaling, which isn't ideal because all instances connect to the same database. Even if an init container fails, the main container still starts, which could lead to issues if there's a failed migration.
2. **Jobs**: These seem to fail because they're immutable—Kubernetes recognizes completed jobs and won't overwrite them during a new deployment, which can lead to failures.
3. **Pre-deploy Requirements**: I need a script that runs immediately when an image digest changes, and only continues with deployment if this script succeeds. The migration job has to run once per image change, must not trigger during scaling events, and a failed migration should halt deployment to maintain the current version.
Some additional context: I use ArgoCD sync waves for different configurations, so the migration job is designated on sync-wave 1, while the main deployments are on sync-wave 2.
2 Answers
Sounds like you’re on the right track with ArgoCD and sync waves. It's a solid approach! The job immutability issue can be a pain, though. I would suggest looking into some of the Enhancements in K8s. Also, using sync options like Replace might help with the job immutability problem. Just watch for those errors!
I totally get your struggle! We handle this with pre-upgrade hooks in Helm, which work pretty well. When we considered moving away from Helm, we tried using a pre-deploy job that made another job a dependency, which is well-documented in Flux. It could be worth checking out!
Yeah, I ran into the same issue with immutability while using sync options. It’s definitely a juggling act!