Full test runs can be slow, but selecting tests from a diff raises questions about shared state, stale mappings, and missed dependencies. How do you make test impact analysis useful without trusting it more than you should?
How do you safely run only tests affected by a diff in CI?
Asked by
Q!Cattail
Answers
Use selection to speed up pull request feedback, not as your only correctness check. Keep a scheduled or main-branch full-suite run as a backstop. If there is no usable baseline mapping yet, running everything is the safer default.
Also make sure newly added tests are included automatically. A missing or stale map should cost you time, not quietly turn a test green by skipping coverage.
Reply to this answer
The payoff depends a lot on how independent the test areas are. If nearly every feature depends on shared core code, a change can legitimately select most of the suite. In a modular project, the same approach may exclude many unrelated tests.
I would measure the selection rate and runtime savings on your own changes before investing much in it. If the selector keeps choosing almost everything, ordinary parallelism may be simpler.
Reply to this answer
For a language with a clear package and dependency graph, recomputing affected packages in CI can be cheap enough to do on every change. Compare that analysis cost with the time saved on tests, and include both tests and executable entry points in the dependency picture.
That approach depends on the graph being complete. Dynamic imports, runtime wiring, and behavior supplied by external dependencies can make static analysis miss relationships.
Reply to this answer
Shallow checkouts make diff handling awkward when the comparison base is not present locally. Instead of assuming a fixed clone depth will contain the right history, consider storing the information needed to resolve changes alongside the test mapping. Key the artifact clearly, and fall back to a full run if the required data is unavailable.
Also treat dependency and lockfile changes as a separate trigger. A code-to-test map may not reveal which tests depend on a library whose behavior changed after an upgrade.
Moxie__4:
That dependency-change gap seems easy to miss. A selector can be accurate for source edits and still be incomplete for changes in the environment the tests run against.
Reply to this answer
Be cautious about treating a green selected run as proof that nothing was missed. One way to build confidence is to replay selections against real changes and check that tests covering changed behavior are selected. Mutation-based checks can help test the mapping itself, though they do not replace a full-suite backstop.
The goal is not to claim perfect coverage from the selector. It is to make its limits visible and keep a reliable route to broader testing.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Q!Cattail:
That separation makes sense. A fast selected run can be useful, as long as something still exercises the whole suite regularly.