How Should an AI Agent Diagnose CI Failures and Detect Flaky Tests?

Reproduction is a useful confidence threshold. If the failure can be reproduced locally with the same inputs, it is probably a code or test issue. If it happens consistently in CI but not locally, treat the environment as a separate hypothesis—look at machine differences, parallelism, network services, timing, and resource limits. If it only appears intermittently on identical revisions, classify it as a flakiness or infrastructure investigation rather than immediately blaming the latest code.

Use the diff to prioritize hypotheses. If configuration, dependency, or pipeline files changed, investigate those first. If the change is isolated to application logic, a genuine regression becomes more likely. Then compare the failing run with a known-good run, including inputs, environment variables, dependency versions, timing, and available resources.

A practical agent could maintain competing hypotheses—code regression, test defect, dependency change, configuration error, infrastructure failure, and resource or timing issue. Start with prior probabilities from historical data, update them using the diff, logs, rerun behavior, and comparisons with successful runs, then report the leading cause along with confidence and the evidence that would change its mind. It should stop when one explanation is supported by multiple independent signals, while explicitly marking uncertain cases for human review.

Before adding a generative system, make sure the pipeline produces useful evidence. Clear error messages, structured test output, artifacts, reproducible commands, and good separation between setup failures and test failures often solve much of the diagnosis problem. An agent should gather and rank evidence rather than invent explanations from incomplete logs.

Start with historical evidence. Check whether the same test has failed before without a meaningful code change, and whether rerunning it made it pass. Repeated failures followed by successful retries are a strong signal of flakiness, although that alone does not prove the test should be fixed. The difficult part is usually collecting reliable labels showing whether past failures were genuinely flaky or real regressions.