I'm really frustrated with how our Continuous Integration (CI) process is affecting our workflow in AI development. We've managed to generate around 2,000 unit tests with AI in just a few days, which initially felt great. However, we've run into a ton of flaky tests and issues despite trying to improve by clarifying our instructions and adjusting sub-agent setups. Now, every pull request feels like a never-ending cycle of fixes due to CI errors. With over 10 PRs a day, we're facing 30 to 40 cycles of 'CI fail, find the error, fix it, and re-run' before anything gets merged. I even tried using Codex CLI to assist with fixes, but it hasn't solved the problem. Honestly, CI is becoming a bigger bottleneck than actual coding or debugging. Are other teams experiencing burnout from CI? Has anyone found effective tools or strategies that keep high-volume AI pipelines running smoothly? Let's share our experiences and solutions.
4 Answers
Have you considered running tests locally before submitting PRs? Waiting for CI results can be a slow feedback loop. It could help speed things up if your team tests their changes first to catch issues earlier.
You might want to consider taking a platform approach to your CI. Treat it like a product in itself instead of a support function. Having dedicated team members focusing solely on CI issues could alleviate some of the bottlenecks you're facing, especially since it seems to impact productivity significantly.
Totally agree, prioritizing CI improvements could help streamline the process.
It’s ironic that you're using AI to tackle these issues, yet it might be creating more problems. I’d recommend reassessing your generated tests. Take a small subset and make sure they're truly necessary and robust. Sometimes less is more; focus on quality over sheer volume with your test coverage to ease CI pressure.
We are improving our tests daily. The initial 2k were carefully planned, but the rapid growth of product changes leads to more CI challenges.
It sounds like the problem isn't just with CI but maybe with your code or tests too. Running 10+ PRs a day isn't that high-paced for many teams, so if CI is failing often, it might point to some underlying issues with your tests or the code itself. Given that you’re heavily using AI for generating tests, it’s not surprising that you’re facing some hiccups. You might want to assess the quality of your tests and how well they integrate with the code you're pushing.
It’s 10 PRs per dev, so the volume is there. But I get your point; there may be a deeper issue with code quality.
Yes, we do run unit tests locally. The issue is more with integration and end-to-end tests being too slow to run locally.