I'm building a POMDP-based CI/CD failure diagnosis agent. The model treats the real cause—such as which component is faulty—as hidden and relies on pipeline observations to choose diagnostic actions. My current categories are application or test defects, pipeline configuration defects, upstream dependency drift, runner resource exhaustion, and authentication or access failures. What important failure states am I missing? I also need the categories to be mutually exclusive and collectively exhaustive, so each failure should map to one primary state without overlapping another category.
4 Answers
The biggest missing category is external platform or service outage. Your source-control provider, package registry, cloud API, container registry, DNS provider, or deployment target can be unavailable even when your code and configuration are correct. I would also separate transient network failures and timeouts from authentication failures, since a valid credential can still fail because a service is unreachable or too slow.
Other useful states are nondeterministic or flaky behavior, clock and environment mismatches, and deployment or policy rejection. A test can pass or fail randomly because of race conditions, while a deployment can be blocked by an approval gate, security policy, quota, expired certificate, or incompatible environment. You may want a separate 'unknown or insufficient evidence' state as well, because a diagnostic agent will not always be able to identify the cause.
Strictly speaking, the proposed states cannot be perfectly mutually exclusive. For example, a bad dependency may cause a test failure, and a misconfigured runner may produce a timeout. A better design is to define one primary fault domain for classification—application, pipeline definition, dependency, execution environment, external service, or access—and store secondary contributing factors separately.
You could make the primary state the earliest actionable cause in the chain. For instance, classify a failed test caused by a broken package as dependency drift rather than application failure, then keep the failed test as an observed symptom.
Consider adding orchestration and scheduling failures: jobs stuck in a queue, unavailable runners, invalid matrices, concurrency or cancellation rules, artifact handoff failures, and jobs skipped because conditions evaluated unexpectedly. These are different from a runner running out of memory or disk space.
A provider outage is especially worth modeling separately because many jobs can fail at once with the same external-service error, while a dependency problem usually affects a particular package, image, or version.