I'm looking for serious advice because our CI/CD pipeline is getting out of hand. It currently takes 47 minutes to run the full test suite, which is severely impacting our deployment speed. We're supposed to deploy multiple times a day, but right now we're lucky to manage one deployment due to long waits for tests to complete and the need to debug failures that seem to work fine locally.
To make matters worse, we've been experiencing around 15-20% false positive failures. Some developers have started just rerunning failed builds until they pass, and others are even bypassing the CI/CD process altogether to push directly to production, which raises obvious concerns.
I've tried to implement parallelized test execution, but that led to new issues with shared states and increased flakiness. Improving test isolation seems like a massive task that I don't have time for at the moment. With management pushing for increased deployment frequency and better developer satisfaction, I need effective strategies or insights into how other teams handle similar situations. Is a 47-minute pipeline typical for a project of our size, or is there something fundamentally off with our approach?
5 Answers
You should definitely look into an iterative approach. Dedicate time in each sprint to addressing flaky tests and make sure developers can’t circumvent the CI/CD process. It's also worth implementing Ignore annotations for any flaky tests, so you can prioritize fixing them later without them blocking your pipeline.
On top of that, consider splitting longer tests into separate jobs to avoid slowing down your main pipeline. This way, you can speed up the dev feedback loop.
If your tests are taking over 40 minutes to run, that's not ideal, but it can happen in larger projects. Try to focus on improving your test suites by removing any tests that frequently fail. Those are just contributing to wasted time. You might also want to look into whether your CI/CD runners can be optimized to run concurrently. Having just one runner can definitely lead to delays.
Honestly, 47 minutes can sometimes be pretty reasonable depending on your application. If you're running end-to-end tests, those can really add up time-wise. You might want to focus on writing more unit tests that run quickly, and make sure your test cases are deterministic. Split your test execution into smaller, more manageable units and run them in parallel wherever you can. If the developers are pushing directly to production, that's a red flag; prevent that from happening at all costs!
It sounds like you might be dealing with some development issues rather than just infrastructure. If your tests are written in a way that doesn't allow for clean parallelization, it's time to take a step back and assess how you're writing tests. Make sure you're segmenting your fast unit tests from slower integration tests, because those should run separately to avoid bottlenecks.
Also, consider skipping unnecessary tests during your pipeline to save time. If your developers are continually rerunning failed tests, focus on fixing those flaky tests instead of allowing workarounds.
I feel your pain! One way to help speed things up is to utilize pre-built Docker images for your dependencies, so you don't waste time building them in every pipeline run. You could also evaluate if any of your E2E tests are taking up too much time. Consider moving them to run in a nightly job instead of during the day.
Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically