I'm working on a CI/CD setup and I've hit a bit of a wall. My repo has numerous features, and when someone enhances just one of them and creates a pull request (PR), I'm unsure if I should run tests for all features or just for the ones that were modified. For instance, I have two scripts, 'script/register_model_a' and 'script/register_model_b', which log metrics and create new model versions in MLFlow. What's the best practice here? Should I set up a folder structure based on each module to detect file changes and decide which feature needs testing, or is it safer to run all tests every time? Thanks for any advice!
4 Answers
For MLFlow, don’t register a new model version on every PR commit. Instead, you can log the metrics on the PR and skip registration. Only do that on merge to the main branch. Organize your repo efficiently and track changes that way. Here are some helpful links:
1. GitLab rules for CI.
2. GitHub actions path filters.
3. Jenkins changeset management.
TL;DR: Run everything every time! Your CI pipeline should have jobs that ideally finish in about 20 minutes for merges or PRs. For example, run as many code scanners as possible in parallel after building. Your integration and system tests should also be parallelized; avoid making tests depend on each other, as this can slow things down. I once optimized a test pack from 30 hours down to just 40 minutes after a lot of tweaks!
You're on the right track! Most teams encounter this issue as their codebase grows. A good solution is to use folder-level ownership and diff-based triggers. Segment each feature into its own module, then check what paths changed in the PR to run the appropriate scripts. Most CI tools can handle this well. However, I suggest doing full runs for main/release branches and only selective runs for PRs to balance speed and safety—this helps avoid lengthy run times, especially with heavy ML tasks.
The easiest route is just to run all tests. If features are truly isolated, you could invest the time to only trigger tests based on what has changed, but that usually pays off only when it becomes cumbersome.
I agree! Plus, since they’re working on features within the same repo, it might be risky to rely entirely on individual tests. Running everything gives a comprehensive safety net.
Good insight! We do the same in my team—only the main branch gets model registrations.