I maintain an open-source agent harness with a verification step. Each changed file maps to a test command, and a hook records PASS or FAIL in a ledger after the command runs. The agent can only finish when the ledger is green.
I found two bugs that incorrectly produced PASS. First, pytest returns exit code 4 when a requested path does not exist and exit code 5 when no tests are collected, but the agent treated the "no tests ran" text in stdout as success. Second, the hook used this shell expression:
EXIT_CODE=$(echo "$INPUT" | jq -r '.tool_response.exit_code // 0')
When the payload omitted exit_code entirely, jq substituted 0, so the hook recorded PASS even though no result had been reported. The fix is to treat a missing exit code as a failure with an explicit reason, and to add a matrix test that fails whenever a row collects zero tests.
For CI, should no collected tests always be considered a failure, or should the wrapper decide?
1 Answer
I’d make both cases red by default. If some matrix shards are intentionally empty, keep that behavior in an explicit allowlist rather than silently converting pytest’s code 5 to success. Also, avoid defaulting a missing field to zero: `.tool_response.exit_code // empty` leaves it empty when absent, so the hook can distinguish “no exit code was provided” from a real 0. A pytest plugin can also normalize the no-tests result, but that policy belongs closer to the test runner than in a generic verification hook.
That distinction is exactly what I needed. None of the shards are legitimately empty, so an empty collection can simply be a real failure for now. Moving any 5-to-0 policy into pytest, if it’s ever needed, also keeps the wrapper from interpreting exit codes itself.