I'm currently using CI that only runs unit tests, but we want to add some basic code-quality gates like coverage above 80% and no new critical issues. The trouble is that every time we try, the pipeline either slows down significantly or starts throwing false positives. I'm looking for effective ways to enforce these standards without having to monitor every pull request manually.
5 Answers
Relying on project coverage alone isn’t enough. Even with 80% coverage, it’s easy to make changes without adequate test coverage afterward. We have great success with CodeCov, where every PR needs to match or exceed the project's total coverage. Plus, implementing tools like Renovate and GitHub CodeQL scans has also been super beneficial.
If you're coding in Python, I highly recommend using `ruff`! It can enforce a broader rule set, including McCabe complexity. My go-to setup includes `pytest` for testing, `hypothesis` for property-based tests, and `ruff` for linting. Additionally, using `pytest-cov` for coverage (don’t forget branch coverage!) helps keep everything in check.
I’d suggest against enforcing strict coverage because it can create unnecessary issues; it's often just a symptom of a bigger problem that requires a cultural fix. To address other quality issues, running a flexible linter can help since most let you tweak which issues break the build. This approach can certainly keep the team accountable without adding extra pressure.
Aiming for coverage thresholds can often be misleading unless paired with meaningful metrics. Just having 80% coverage doesn’t guarantee safety, as developers might write tests just to meet the number. It's better to look into mutation testing or using tools like SonarQube to monitor code smells and linting with fail conditions based on your team's noise level.
It’s not overrated, though! Configuration tweaks for coverage can help. For instance, excluding generated code or trivial getters can boost your percentage over 90% while acknowledging some lines are simply hard to cover.
Try initially setting the coverage limit to your current coverage level and slowly increasing it over time. This way, your team can grow into the standards without feeling overwhelmed. It’s crucial in a team environment, particularly with a growing codebase!
I've explored mutation testing, but honestly, it doesn’t feel effective unless in high-stakes projects like healthcare or aerospace. The tests take ages to run and sometimes find issues that are way more costly to write tests for than just dealing with the potential bugs. It often feels like we could achieve better results through mentoring developers on test writing instead.