How do you handle a slow integration test with trunk-based development?

November 16, 2026 4 answers
Asked by Leni

We want to merge changes straight to main, but our full end-to-end DAG test takes over an hour and is costly to run on every pull request. Could we run faster checks on each change and save the full test for the end of the sprint?

Answers

4j~canvaswhistle_638

I'd split the checks by what they need to catch. Run unit tests, schema and contract checks, and a small critical-path DAG run on each pull request. Then run the full DAG on main nightly and again against the exact commit you plan to release.

Make the full run produce a clear pass or fail rather than relying on someone to inspect the output manually. A nightly failure should be treated as a priority, otherwise the broken state can hang around until release day.

Leni:

That sounds more manageable than running the whole DAG on every pull request. The nightly run would also give us a smaller set of changes to investigate.

Reply to this answer
Bexlin+9u

I wouldn't wait until the end of the sprint for the first full run. If it fails, you have a whole sprint of changes to investigate, and the failure could hold up the release while you work out which change caused it.

Nightly is a reasonable compromise, but it means main can be broken between runs. Have a plan to fix or revert a bad change quickly. You could also pause merges when the latest full run fails, if the team is prepared to treat that as a priority.

Reply to this answer
_d93~claspmortar.5

Another option is a merge queue that batches approved changes and runs the full test on the combined batch before it reaches main. That can reduce the number of expensive runs while keeping the slow test as a merge gate.

The tradeoff is waiting for the queue. If a batch fails, you'll also need to work out which change caused it before those changes can merge.

Reply to this answer
_v83!cachewrench.5

Main doesn't have to be the rollback target if you keep track of the commit and artifact for each verified release. Roll back by redeploying the last known good artifact, and make sure any hotfix is tested against the release it will actually ship from.

That said, release tags don't tell you whether the latest main is healthy. They solve a different problem from frequent integration testing.

d.297~mapgrommet:

A hotfix made from a previous release also needs to make its way back to main, so it doesn't get lost in a later release.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions