I'm curious about the practical routine people follow when code works locally, gets pushed, and then fails in the CI/CD pipeline. If the failure can't be reproduced on a developer machine, what do you check first, and how do you decide whether to keep investigating, rerun it, or postpone the release? I'm especially interested in the realistic end-of-day version rather than the textbook process. Also, with lockfiles and containers being common now, is this still a major source of trouble in 2026, or have those tools mostly solved it?
4 Answers
At 5 p.m., I’m not forcing a deployment through a red pipeline unless there’s clear evidence the failure is unrelated and already understood. I’ll capture the logs, check whether the previous known-good commit fails the same way, and classify it as a code failure, flaky test, runner/resource problem, dependency change, or infrastructure issue. If the failure is real or unclear, the release waits. Spending an hour guessing and then creating an incident is usually worse than picking it up with a clear head the next morning.
A clean clone and build is one of the fastest checks. A surprising number of failures come from something present on a developer’s machine but missing from the repository or deployment package: an undeclared dependency, generated file, stale cache, or project file that wasn’t included in publishing. If that doesn’t explain it, compare the CI environment with production and investigate timezone, locale, permissions, kernel behavior, service readiness, and filesystem differences. CI being green matters more than one developer’s local result because it is closer to the environment that will actually run the code.
Treat CI as the source of truth and don’t deploy until the failure is understood. Start with the job logs and metadata: commit, runner image, runtime and dependency versions, cache state, test seed, timezone, locale, database version, and resource limits. Rerun the specific failing test with a clean build and maximum verbosity. If it still fails, reproduce those same constraints locally with a fresh clone, clean install, matching container, test order, environment, and CPU limits. Once fixed, turn the case into a regression test or a clearer preflight check. Lockfiles and containers reduce dependency drift, but they don’t eliminate timing, state leakage, resource limits, database ordering, or environment differences.
My first move is usually to rerun only the failing test in CI. If it passes alone but fails in the full suite, I look for test-order dependence, shared fixtures, database rows that weren’t cleaned up, or a race. I also want the exact random seed and test order so I can replay the same run locally. A live debugger attached to a CI process sounds useful in theory, but many flaky failures disappear on rerun or happen after the process has already exited, so deterministic replay and good diagnostics are often more valuable.
That matches the distinction I was looking for. I tend to get stuck adding log statements and comparing environments, so capturing the seed, test order, and exact runner constraints seems more useful than immediately trying random local changes.