I'm researching a small CI diagnosis agent and want to understand how experienced engineers investigate integration-test failures when the logs don't point to an obvious cause. What do you check first: the stack trace, database and service availability, recent code changes, the last successful build, test or database state, prior failure history, or simply rerunning the test? More importantly, how do you decide what evidence to gather next? For example, if a recent change touches the code involved in the failing test, what would you verify before concluding that change caused the failure? I'm especially interested in practical debugging habits, including how you distinguish a code or data problem from a flaky test, dependency issue, or CI environment change.
4 Answers
Don't assume the application code is the only suspect. A container-runtime or platform update can break a previously stable pipeline, and resource contention on a different machine can cause intermittent failures. Check the health of databases and external services, inspect test and database state, and look for changes in the larger execution environment. If the failure only happens in CI, prioritize environment, dependency, timing, and configuration differences; if it also fails locally in a clean setup, confidence increases that the problem is in the code, test, or data.
The most valuable comparison is often between the environments rather than just between source revisions. Make the CI setup reproducible locally when possible, including tool versions, services, fixtures, and dependency installation. A local failure in that clean setup points toward the test, application, or data. A local pass but repeated CI failure points toward differences such as machine resources, networking, credentials, service timing, image versions, or cache contents.
A useful process is to form a small hypothesis, gather evidence that could disprove it, and change one variable at a time. If a recent code change looks suspicious, compare it with the last successful revision, rerun against the same inputs, inspect whether the failure is deterministic, and verify that the environment is unchanged. A correlation with the latest commit isn't enough by itself—timing, dependency updates, test ordering, stale data, and infrastructure changes can produce the same pattern.
Start by ruling out the cheap, broad possibilities. Rerun the test while checking the logs and dependent services in parallel. A pass on rerun suggests flakiness or an environmental problem, while a consistent failure gives you something more reliable to reproduce. Then compare the failing run with the last green build, paying attention not only to application changes but also to OS images, system packages, tool versions, containers, caches, and configuration. If it reproduces consistently, recreate a clean environment locally that resembles CI, clear dependencies and caches, and use the CI machine's interactive debugging access if available.
That distinction is helpful. When reproducing locally, I’d want to know whether the setup is genuinely clean and CI-like; otherwise a local pass doesn't tell me much about the original failure.