We use trunk-based development with one main branch deployed to staging first and then production after validation. Alembic migrations run as part of updating the service.
We have two recurring problems. First, if migration A is merged and fails or causes trouble in staging, a later migration B depends on it in Alembic's revision history. Even if B is unrelated, it cannot run in production until A is fixed. Second, migrations for two independent features can be merged in sequence, but if feature B is ready while feature A is not, deploying the service to production still applies both migrations because Alembic follows the complete revision chain.
In both cases, the migration history couples unrelated feature work. Has anyone solved this cleanly in production, and what tradeoffs did you accept?
3 Answers
Prevent bad migrations before they touch shared staging. For every change, run the complete migration sequence against a temporary database in CI, preferably both from an empty database and from a realistic snapshot. Add tests for data migrations and review schema changes separately. A protected merge gate is much cheaper than recovering a shared environment after a broken migration.
For more realistic testing, use an isolated database or short-lived environment per branch or pull request. This avoids having unrelated work contend for one staging database. It does cost more infrastructure, but it gives each change a clean migration history.
The usual trunk-based answer is that anything merged to main must be safe to deploy, including its database migration. Keep unfinished behavior behind feature flags rather than trying to hold back its schema change. Use an expand-and-contract approach: add new tables or nullable columns, deploy code that supports both old and new schemas, backfill data, switch reads and writes, and only remove the old structure later.
That means feature A’s migration may reach production while feature A remains disabled, which is generally acceptable. What should not reach main is a migration that is unsafe to apply in production.
This also means the staging-to-production promotion should use the exact artifact or commit that passed staging, rather than whatever newer state happens to be on the branch.
Alembic can represent multiple heads and later merge them, but using migration branches as a release mechanism usually just shifts the complexity elsewhere. You still need to decide how application code behaves when only part of the schema exists, and skipping a parent migration can leave environments with incompatible histories.
If a migration has already reached a shared environment, the safer general strategy is to fix forward with a new migration. If it failed before applying, you may be able to correct or replace it according to your migration policy, but once production data is involved, rewriting history is risky. For an intentionally unwanted change, add a compensating migration rather than pretending the original never existed. Also promote immutable builds or tags, not the moving tip of the branch.
If full environments are too expensive, a temporary database container in CI still catches most schema-ordering and migration-syntax problems. Reserve snapshot-based testing for migrations that transform or repair real data.