I'm researching a small agent that could help diagnose CI failures, but I want to understand how experienced engineers approach the problem in practice. Suppose an integration test fails in CI and the logs or stack trace don't immediately explain why. What do you investigate first: rerunning the test, checking databases and dependent services, reviewing recent changes, comparing with the last successful build, inspecting test data, reviewing earlier failures, or reproducing it locally? More importantly, how do you choose the next investigation step? For example, if a recent change touched code used by the failing test, what additional evidence would you want before blaming that change? I'm especially interested in practical debugging habits rather than an idealized CI/CD workflow.
4 Answers
Use a systematic elimination process and keep track of what each experiment tells you. Check whether the test is flaky, whether dependencies are healthy, whether the same inputs and database state are being used, and whether recent infrastructure changes line up with the first failure. If needed, isolate or replace one integration point at a time with mocks or known-good services. Look for patterns across previous failures instead of treating every red build as an unrelated incident.
If the failure keeps happening, compare the failed build with the last known-good one. Don’t only compare the application commit—look at the runner image, operating-system packages, language or toolchain versions, container images, caches, configuration, and database state. Environment changes are common causes, including updates to the CI platform or Docker tooling, even when no application code changed.
Start by ruling out the cheapest and broadest possibilities. Rerun the test while checking the logs and external dependencies in parallel. The result of a rerun is useful by itself: a pass suggests flakiness or environmental instability, while a repeatable failure gives you a stronger signal to work from. Also compare the result with a local run, but treat that as evidence rather than proof.
For a reproducible failure, make the local setup as close to CI as possible: use the same dependency versions, a clean environment, equivalent data, and ideally the same container or runner image. If it fails locally under those conditions, code, test logic, or data becomes more likely. If it only fails in CI, investigate resource contention, timing, network access, credentials, service configuration, machine differences, and cache behavior. Access to the failed build machine can also reveal details that aren’t present in the normal logs.
A local failure is useful evidence, but it doesn’t automatically prove the application is at fault. The local environment may differ in timing, hardware, service versions, or data. Likewise, a CI-only failure doesn’t rule out a code defect; it may just require concurrency or a particular ordering to expose it.
A change touching the failing function is only a correlation. I’d want the failure to begin after that change, a stack trace or assertion consistent with its behavior, reproduction with the same inputs, and ideally a comparison or rollback showing that the failure disappears. Otherwise, infrastructure, data, timing, and dependency changes remain plausible alternatives.