I'm building a system to estimate whether committed code was created with assistance from AI coding tools. At the moment, I'm looking at Git and commit-level signals such as AI-related trailers, commit metadata, lines changed, files touched, addition/deletion patterns, and commit frequency.
The main challenge is confidence and calibration. A commit with hundreds of new lines may be completely legitimate, and developers can remove or alter metadata that might reveal AI assistance. Once code leaves the development environment and reaches version control, much of its provenance may be gone.
I'm trying to determine whether this should be treated as probabilistic risk scoring rather than a binary AI-versus-human classification. Which Git or CI signals are genuinely useful, how should thresholds be calibrated, and are there better ways to preserve provenance earlier in the workflow? I'm especially interested in repository- or pipeline-level approaches, research, open-source projects, and practical experience. I don't expect perfect detection; a measurable estimate with known false-positive and false-negative rates would already be valuable.
4 Answers
I’d frame this as risk scoring, not detection. Build a labeled evaluation set containing ordinary human changes, openly AI-assisted changes, and mixed workflows, then measure precision, recall, calibration, and false-positive rates separately for different repositories and change types. Use added lines and surrounding context rather than raw diff size, and be cautious with comment-to-code ratios or commit frequency because those vary heavily by team and language.
Also consider scoring what CI can directly verify: tests, static analysis, security checks, review quality, and whether the change satisfies its requirements. Authorship inference has less evidence and is easy to evade, while verified behavior remains observable even when the code was produced with an AI tool.
A co-author trailer or other agent metadata can be a useful positive signal when it is present, but its absence tells you very little because it can be omitted or removed. If provenance matters, capture it earlier: record tool usage through the development environment or an approved agent service, sign the resulting attestation, and pass that evidence into CI. That gives you an auditable claim instead of trying to reconstruct authorship from the final diff.
I’d avoid treating large diffs or total lines changed as strong evidence. Repository setup patterns may be more informative in aggregate: whether files look manually initialized or generated from a template, how package metadata is populated, whether standard project fields are missing, and whether required documentation or test details are filled in unusually consistently. These are weak clues individually, so combine them into a score and validate them against known examples rather than using any one as a rule.
The most important limitation is that authorship becomes very difficult to infer once a skilled developer has reviewed, edited, and integrated the output. AI assistance is a tool, and the final code may be indistinguishable from code written entirely by a person. Treat any result as an uncertain signal rather than proof, and consider whether the actual goal is code quality, security, policy compliance, or provenance.
That makes sense, but I’m a junior developer and this is an assigned project. I still need to build something reasonably useful, even if perfect detection is impossible. I’m hoping to reach a measurable accuracy level and document the limitations clearly.
Would you compare historical repositories created before widespread AI tools with repositories known to use them? I’m also unsure whether a model such as CodeBERT is necessary or whether simpler engineered features would be easier to calibrate.