I'm building an agent that investigates failed CI runs. Given the pipeline result, logs, test output, repository changes, and other run metadata, I want it to reason about possible causes rather than simply report the first error it finds.
How do experienced engineers approach a failed CI run? What initial hypotheses do they form, such as a real code regression, a configuration problem, an infrastructure issue, or a flaky test? How do they decide which evidence to examine next and weigh competing explanations? In particular, how can the agent distinguish a test that intermittently fails on unchanged code from a legitimate failure caused by a recent change?
I'm also interested in the stopping point: what evidence makes an engineer confident enough to call a cause, and when should the agent continue gathering information or report uncertainty?
3 Answers
Before adding an AI layer, make sure the pipeline produces useful evidence. Clear step boundaries, structured test results, preserved artifacts, dependency versions, environment details, and actionable error messages often make diagnosis much easier. If the logs are ambiguous, a generative model may only produce a convincing explanation for missing information.
The difficult part is usually not writing the reasoning rules; it is obtaining reliable labels for past failures. Most teams do not consistently record whether a failure was a flaky test, a code regression, a configuration issue, or an infrastructure problem. Historical outcomes and explicit human confirmations would make a diagnostic model much more useful.
A useful first check is whether this exact test has failed before without a relevant code change. Historical run data can reveal that pattern: compare the same job across many pipelines, especially cases where a retry passed. Repeated failures followed by successful retries are strong evidence of flakiness, although a retry alone is not definitive.
Then rank hypotheses by what changed. A modified dependency or pipeline configuration points toward the environment or setup; a change in the implementation or test points more toward a real regression. The agent could turn this into a sequence of checks instead of trying to infer everything from one log line.
Reproducibility is one of the strongest signals. If the failure reproduces locally with the same inputs and setup, a code or test defect becomes much more likely. If it cannot be reproduced locally but occurs consistently in CI, investigate differences in the runner, timing, dependencies, permissions, network, and other environmental factors.
For the agent, the result should probably be a confidence-ranked diagnosis rather than a forced binary answer. It can report the leading cause, supporting evidence, contradictory evidence, and the next experiment that would distinguish between the remaining possibilities.
That makes sense. I was especially trying to understand how to represent those hypotheses and decide when the evidence is strong enough to stop investigating.