Best Practices for Handling Database Migrations in Microservices Production

In my project, we implemented a separate migration job with FluxCD managing the process. Here’s a quick workflow: The CI/CD pipeline builds and pushes the new application image, then Flux monitors the registry and executes the migration job first. Once migrations are completed, the application restarts with the new image. It's a clean way to avoid issues where migration might conflict with ongoing deployments.

We created a dedicated migration service that runs as a prerequisite to our application pods. This way, anytime there's a schema change, the CI/CD pipeline automatically checks out the main branch, sets up a Postgres instance, runs the necessary migrations, and then proceeds with the application deployment. This method helps us catch any breaking migrations early in the CI process.

We handle migrations with a dedicated job that starts up, performs the migration, and shuts down afterward, which is managed by ArgoCD. This method prevents conflicts from occurring between different pods trying to access the database concurrently during migrations.

A solid option is to run migrations directly from your application itself. I find it super effective to have migrations tested alongside unit and functional tests. It keeps things simple and you stay away from relying on infrastructure-specific elements outside of your codebase. I've seen mixed opinions on CI/CD handling migrations, but I lean towards the application handling that at startup instead.