I'm learning Kubernetes with kind and have five Node and Express services, each with its own Postgres database and Drizzle migrations. Should migrations run in an init container or a separate Job? Would the choice change for production?
How should I run database migrations in a local kind cluster?
Asked by
7n~clampwheel_526
Answers
I'd start with a separate Kubernetes Job for each service's migrations. Run it as part of the release process, wait for it to complete, and then roll out that service. The Job can use an image containing the migration files and get the database connection details from a Secret.
A Job can retry after a failure, so the migration process still needs to handle retries safely. Drizzle tracks applied migrations, but that alone shouldn't be treated as protection against two migration processes running concurrently. Make sure your release process doesn't start overlapping migration Jobs for the same database.
Reply to this answer
An init container is a reasonable gate, but I wouldn't put the migration itself there if the service can have multiple replicas. Each pod runs its own init container, including on replacement or restart, so several pods could try to migrate the same database at once.
The gate could wait for the release's migration Job to complete before starting the app. Give the Job a release-specific name or otherwise make sure the gate is checking the right Job. If it uses the Kubernetes API, it also needs appropriate permissions.
y.683~teaspoon:
Waiting in the release process before updating the Deployment is another option. An init-container gate is useful if you also want each new pod to check that condition.
Reply to this answer
For production, think beyond just getting the Job to finish. Old and new application pods may overlap during a rollout, so a destructive schema change can break the old version even if the migration succeeds.
Where possible, make schema changes compatible with both versions first, deploy the application changes, and remove old columns or constraints in a later release. Have a recovery plan and a database backup strategy before running changes that are hard to reverse.
Reply to this answer
If you use Helm, a hook can run a migration Job as part of a release. That's handy for automating the ordering, but it's optional. For a learning project, a Job and an explicit wait are easier to inspect and debug before adding another layer.
Reply to this answer
Add your answer
Answers are reviewed before publication.
7n~clampwheel_526:
So locally I can apply the Job, wait for it to finish, and then apply the service deployment?
u+Ravik.418:
Yes. You can use kubectl wait for the Job's Complete condition with a timeout, and check the Job logs if it fails or times out.