We're struggling with our testing pipeline which is bogging down our productivity. We have a Selenium test suite with around 650 tests that runs on every pull request, averaging about 40 minutes per run, but it sometimes takes an hour. The biggest issue is flakiness—about 8 to 12 tests fail on each run, always different ones. Our developers have resorted to simply clicking 'rerun' and grabbing coffee while they wait.
6 Answers
Sounds like you're in a tough spot! One big factor could be management's mindset. If they're pushing for constant shipping with flaky tests, it’s time to have a serious talk. In my experience, management needs to acknowledge that if the pipeline is failing often, it’s dragging down overall morale. That said, you might consider temporarily disabling some of those flaky E2E tests while you fix the underlying issues, especially if they aren’t catching real bugs.
Agreed, it’s crucial to put focus where it’s needed! You can't keep shipping flawed code just for the sake of speed.
Have you checked how many of those 650 tests are actually necessary? It's worth reviewing each one to see if they're still relevant or if any can be removed. You might find a lot of old tests that don’t apply anymore which could clear up some of the noise.
You might need to tackle the flakiness directly. Are your tests designed to handle asynchronous actions properly? If they aren’t, that can lead to unreliable results. Switching to a more robust framework like Cypress could potentially solve some of these issues too.
What about enabling auto-retries for those flaky tests? It could save some hassle while you're working on making the tests more stable. Sometimes just a little tweak like that can help mitigate the pain temporarily.
Budget could also play a role here. If you're running tests on slow machines, it might be worth talking to management about investing in better resources. Good hardware can make a huge difference in speed and reliability, especially when running tests in parallel.
True! But it'll only work if the management sees the value in speeding things up.
It sounds like the existing strategy might not be optimal for your project. Instead of running all tests on every PR, consider running only a subset of key tests. This way, you're getting feedback quicker and can avoid the bottleneck while still maintaining quality.
Totally! Targeting critical paths can reduce the load while still ensuring quality.
Exactly! If the tests aren’t reliable, they become a bigger problem than just running them. Management needs to take notice of the productivity drop.