We're facing a major slowdown in our CI/CD pipeline with around 800 automated tests that currently average 45 minutes to run, sometimes even over an hour when it's slow. The real issue is a lot of these tests are flaky, with about 5 to 10 different tests failing randomly on each run. This has led to developers simply rerunning the pipeline hoping for a different result, which completely undermines the testing purpose. We want to implement multiple daily deployments, but the QA stage has become a major bottleneck. We've tried parallelizing the tests, but have hit resource limits and being selective about what runs on each pull request reveals the risk of missing important tests. We're stuck in this frustrating cycle of slow and unreliable tests. Has anyone found a good way to make tests run faster, minimize random failures, and ensure that real issues are caught?
3 Answers
I totally understand your pain. Flaky tests are a huge problem as they erode trust in the testing process. My team tackled this by tracking flaky tests for a week, either fixing or removing the worst ones. After that, it felt much more manageable. Consider using an APM tool to see where your pipeline resources are being consumed. This can help identify bottlenecks without breaking the bank on observability tools.
Start by removing flaky tests instead of just rerunning them. If a test fails randomly, it's doing more harm than good. Instead, analyze what's causing the slowdown; you might have some tests that are hitting your database unnecessarily or waiting on mock network calls, which can seriously bog things down.
You might want to split your tests based on type. Run fast unit tests on every pull request and reserve the slower, flakier tests for nightly runs. Also, implementing retries for flaky tests and setting up better isolation can reduce those random failures significantly.
Isn't that the goal? To create robust tests that don't fall apart under slight changes? Better isolation should really help with that.