I maintain an open-source agent harness with a verification step. Each changed path maps to a test command, and a hook records PASS or FAIL in a ledger that determines whether the task can be marked complete.
Three entries pointed to test files that had been moved or split into sibling files. Pytest returned exit code 4 for nonexistent paths and exit code 5 when filtering with -k collected no tests—neither case returned 0.
The verification still became green for two reasons. First, the agent interpreted "no tests ran" in stdout as success. Second, the hook extracted the exit code like this:
EXIT_CODE=$(echo "$INPUT" | jq -r '.tool_response.exit_code // 0')
When the payload did not contain an exit_code field, jq substituted 0, and the hook recorded PASS. The fix is to treat a missing exit code as an explicit failure with a reason, and to add a matrix test that fails whenever any entry collects zero tests.
In CI, should zero collected tests always be treated as a failure, or should the wrapper around pytest decide?
1 Answer
I’d make zero collected tests red by default. If a matrix shard is intentionally allowed to collect nothing, use an explicit allowlist for those shards rather than silently treating pytest’s exit code as success.
Also, `.tool_response.exit_code // empty` is safer than defaulting to zero. It leaves the value empty when the field is absent, so the hook can distinguish a real exit code of 0 from missing data. The conversion from pytest’s code 5 to success belongs in pytest configuration or an explicit policy layer, not in a generic result parser.
That matches the fix. None of the shards here are legitimately empty, so any zero-test result is a real failure. I also hadn’t considered handling pytest’s code 5 through a dedicated pytest configuration instead of making the wrapper interpret it.