I've been building an ERP and a B2B application with SvelteKit using frontier AI models. My usual workflow is: define the specification, have the model question and refine it, create a plan, implement it, and then fix any issues. For the past two or three months, I've written very little code manually, and the production results have been surprisingly solid.
The trouble started when I used the same models for broad, codebase-wide reviews. They identified many potential problems, and I had other models validate those findings before applying the suggested fixes. I even built a small multi-model review pipeline.
When I checked the findings with end-to-end tests, though, most turned out to be imaginary, based on incorrect assumptions, or irrelevant to the way the application actually behaved. Several models would confidently agree on the same incorrect issue, but often admitted it was wrong once I challenged the assumption.
I'd rate these models close to 100/100 for specifications, planning, implementation, and debugging, but perhaps 10/100 for broad code review. My current takeaway is that model consensus is weak evidence unless it's backed by executable verification. For now, I'm pausing AI code reviews and treating end-to-end tests as the source of truth.
Has anyone found a review workflow that keeps false positives under control?
4 Answers
Treat AI review comments as hypotheses, not confirmed findings. Consensus between models is a weak signal unless each issue is tied to a failing test, a static-analysis rule, or a documented requirement.
A more reliable workflow is to review one pull request or module at a time, provide the relevant project conventions, require the model to cite the exact code path, and ask for a minimal reproduction. If it can’t explain how to reproduce the problem, discard the finding. This eliminates a lot of hallucinated issues.
One alternative is to review the development session as well as the resulting code. Share the reasoning, decisions, alternatives, and tradeoffs, then examine whether the implementation follows from those choices. That can expose misunderstandings earlier than a detached codebase scan, although it shouldn’t replace tests or direct code inspection.
The model matters, but the review process matters more. Some models are noticeably better at spotting subtle problems than others, so it’s worth comparing them on the same pull requests. Still, even a strong model should have to justify each finding against the actual application behavior rather than merely agreeing with another model.
Separate finding possible issues from accepting them. Let the model generate hypotheses, but require every accepted finding to identify the violated requirement or invariant, include a minimal test that fails before the fix and passes afterward, and reproduce the user-visible behavior on the same browser path.
It’s also useful to weaken or remove the proposed fix and confirm that the test catches the regression. Otherwise, a green end-to-end suite may not actually prove the review claim.
I’m not convinced reviewing the session alone would catch bugs or hallucinations. I’d still want the code paths and behavior verified directly, and having the model run the end-to-end checks would probably be slower than doing that part myself.