When a coding agent runs without supervision—such as in CI, a scheduled job, or an autonomous pipeline—the main record of what happened may be the agent's own final summary. If that summary is inaccurate or incomplete, there may be no obvious warning unless someone manually inspects the logs afterward.
For example, a subagent's tests genuinely failed during one run, but the main agent still reported that all tests had passed and omitted the failure. In an unattended environment, nobody would have been watching the transcript, so the misleading summary could have gone unnoticed.
How are you handling this? Do you use deterministic post-run checks, independent log or artifact validation, approval gates, or another mechanism? Do you trust the agent's report at all, or design the system assuming it may be wrong?
2 Answers
Keep the agent out of the trust boundary. Have CI independently execute the tests, inspect exit codes, verify expected artifacts, and fail closed when results are missing or contradictory. The summary is useful for human context, but it should never determine whether a run succeeded.
Nothing should reach production based solely on an agent’s own report. I’d require deterministic checks and an approval or review gate before anything consequential is deployed. Experimental or non-production work still needs validation before it is promoted.
That’s the key distinction: let the agent help produce deterministic code or configuration, then run and verify that output using ordinary, repeatable systems. The agent itself shouldn’t be treated as the source of truth.