I'm designing a local CLI harness for agent-assisted development with distributed repositories, Java services, Kafka contracts, and shared Ansible deployment code. The goal is a deterministic cleanroom that limits common AI failure modes without allowing an autonomous agent to judge its own work.
We have strict API and token limits, so recursive loops and loading entire codebases are out. The team is also skeptical of AI-generated changes, and developers must remain in control: the harness runs locally, developers inspect atomic commits and push manually, and every merge request requires at least two human approvals.
The main risks are circular validation, where the model implements a mistaken assumption and writes tests that merely confirm it; loss of compiler and linter friction; and cross-repository contract failures where one service compiles but breaks Kafka consumers or shared Ansible expectations.
The proposed supervisor wraps git, mise, and the coding agent. mise pins tool versions and credentials. Versioned Python skills check individual physical invariants using binary pass/fail results without consuming model tokens. Test directories are protected with SHA-256 manifests, edits outside the declared path scope abort the run, and changes to tests require an explicit bypass. Output is aggressively pruned, and the loop stops after three failed attempts.
I'd like feedback on three areas: whether binary results plus concise diagnostics are enough for an agent to correct itself; the leanest way to check downstream compatibility locally without expensive integration builds; and whether presenting the system as deterministic safety and anti-cheat infrastructure, rather than a productivity booster, helps skeptical engineers trust it. I'm especially interested in edge cases and blind spots.
3 Answers
The overall direction makes sense, but binary results alone are usually too little feedback. Keep the oracle’s decision binary while returning a tiny, structured diagnostic envelope: check ID, failure category, affected file and line, expected invariant, observed value, and a stable remediation hint. That gives the agent actionable information without dumping stack traces or spending many tokens. Prefer compiler-native diagnostics and structured reports such as JUnit XML, JSON Schema, Kafka schema diffs, or Ansible-lint output over regex parsing of terminal text. Regex can be a last-resort adapter, but it should be versioned and tested against representative failures.
The anti-cheat controls will probably earn more trust than claims about speed, provided they are transparent and not annoying. Log every command, changed path, oracle version, manifest result, and circuit-breaker decision so a developer can audit what happened. Keep humans responsible for requirements and review; the harness should only enforce boundaries and collect evidence. Also test the harness itself with intentional violations: modified tests, symlink escapes, generated files outside scope, stale manifests, missing tools, and a service change that breaks a pinned consumer. Deterministic safeguards are valuable, but they can still encode the wrong invariant, so periodically review the oracle library with domain owners.
For cross-repository safety, build a small compatibility matrix instead of running every integration suite. Pin and publish the consumed interfaces—Kafka schemas, API specifications, generated client models, Ansible variable contracts, and supported version ranges—and run local diff checks against the relevant downstream snapshots. Classify changes as compatible, conditionally compatible, or breaking. Add a few consumer-driven contract fixtures for the highest-risk paths. This catches renamed fields, type changes, removed defaults, and variable-boundary violations cheaply, while reserving full integration builds for changes that pass the first gate but touch a high-risk contract.
A useful extra safeguard is to require an explicit contract-impact declaration in the task scope. If the changed files or generated interfaces disagree with that declaration, stop rather than trying to infer safety from compilation alone.
That distinction is useful: a binary verdict for governance, plus a compact machine-readable explanation for correction. I’ll prioritize structured compiler and test reports and keep raw output available only for human inspection.