I'm building a system to estimate whether committed code was created with help from AI coding tools. Right now I'm looking at Git and commit-level signals such as AI-related trailers, commit metadata, lines changed, file counts, addition/deletion patterns, and commit frequency. The challenge is confidence: a large commit can be completely legitimate, and developers can remove or alter metadata before pushing, so much of the original provenance may disappear between the IDE and the repository. Would it be more practical to treat this as a probabilistic risk score instead of an AI-versus-human classification? Which repository or pipeline signals have proven useful, how should thresholds and false-positive rates be calibrated, and are there better ways to preserve provenance earlier in the workflow? I'm especially interested in research, open-source projects, and approaches that work at the repository or CI/CD level rather than relying only on source-code style analysis.
4 Answers
The main limitation is that authorship becomes nearly impossible to distinguish once a skilled developer reviews, edits, and integrates the output. An AI detector can only estimate risk from incomplete evidence, not reliably prove who wrote the code. I would frame the result as a confidence score and clearly measure it against a labeled test set, rather than promising a fixed accuracy such as 80 percent.
If provenance matters, preserve it before the code reaches Git. For example, record whether an approved coding assistant was used, capture tool and model identifiers where policy allows, and attach signed or tamper-evident metadata to the change or review event. CI can then verify provenance claims instead of trying to reconstruct them from a diff. This still requires a policy for untracked tools and local edits, but it is more defensible than inferring authorship from line counts.
Commit size by itself is too noisy to be useful. If you experiment with heuristics, focus on added lines and combine several weak signals: unusually verbose comments, project files that look manually reconstructed instead of generated by the normal tooling, missing repository metadata, unusual dependency setup, and optional AI co-author trailers. These should be treated as features in a model, not decisive rules. Calibrate them using known human-written and AI-assisted commits from comparable projects, then choose thresholds based on the cost of false positives and false negatives.
Be careful about turning this into a hard gate. Developers who know the rules will optimize their commits to avoid detection, and a detector based on commit shape may end up rewarding code that merely looks less machine-generated. A more useful CI outcome may be verification: test coverage, static analysis, security checks, dependency review, and evidence that a human reviewed the change. Those are observable properties and remain useful regardless of how the code was produced.
A historical comparison can help, but pre-AI repositories are not automatically a perfect human baseline. Project age, language, team practices, templates, and formatting changes can all shift those ratios, so validation data should match the repositories where the system will actually run.