Part 1 of 4 in the Local AI Code Review series
When writing code with AI and Spec Driven Development (SDD), I have found that there are still mistakes in spite of good specs. No matter how detailed or concise the spec, things were missed. When searching for a way to have “self healing” reviews to validate implementations, I came across many methods. One of the methods that seemed to work better than others was a utility from imbue-ai call “verify everything” (vet) [1]. This utility used python scripts and LLM as a judge to review diffs and score them to validate the spec was implemented correctly.
After a few weeks of regular use, vet caught real issues: logic errors that would have slipped through, silent failure patterns buried in error-handling code, and missed tests. Not every flagged issue was valid, but the signal-to-noise ratio was good enough that running vet on every commit felt worth the overhead
Then the session limits started appearing.
Vet’s judgment step, which decides whether a flagged issue is actually worth surfacing, runs against a cloud LLM by default (Anthropic or OpenAi). However, vet can use the existing agent if the cloud provider is not available. Using cloud or agent API connection for a judge comes with tradeoffs. The most common is session token limits, which often stopps work once hit.
Rather than accepting the limit as a fixed constraint, the more interesting question was whether a locally-run model could serve as the judge, eliminate the session limits, and maintain the quality that made vet useful in the first place. This article covers what vet is, how its LLM judge step works, and why that architecture makes a local replacement worth exploring. That investigation is what this series documents.
The vet Pipeline: From Diff to Developer
The flow from a code change to a surfaced issue passes through several stages. A developer commits code or opens a pull request. Vet’s issue identifiers analyze the diff and produce a set of candidate issues. Each candidate then passes through a judgment function that acts as a filter.
The judge receives four inputs: the unified diff, the commit message, the issue description, and a guide that defines what the issue type is meant to detect. It returns a verdict, pass or fail. Only issues that pass the verdict reach the developer.
The asymmetry here matters. False positives, invalid issues surfaced to the developer, erode trust over time. Enough noise and reviewers start dismissing vet’s output without reading it. False negatives, valid issues filtered out, reduce coverage and defeat the tool’s purpose. The system is calibrated to avoid false positives first, which means the judge is intentionally strict.
The token cost concentrates in the judgment step. Every candidate issue requires a separate call to the cloud LLM, with roughly 1,500 to 2,500 tokens of context per call. On an active codebase or large changes, this adds up quickly. The session limit is not a failure of the tool; it is the natural consequence of using a cloud API for every individual judgment call.
The Six-Question Rubric
The judge does not form a general impression of whether an issue seems valid. It answers six specific yes-or-no questions, each targeting a distinct way an issue can be flawed.
q1: Is the issue grounded in specific code, rather than in the absence of something? An issue that says “there is no input validation here” is flagging missing code, not a defect in the code that exists. This question filters out absence-of-information issues, which cannot be verified from the diff alone.
q2: Does the issue avoid speculating about how code might be used, without evidence of that usage in the codebase? A concern like “this could be called with untrusted input” is speculation unless the code diff shows the actual function call. This question filters out issues that project hypothetical usage scenarios onto code the diff does not demonstrate being misused.
q3: Is the severity reasonable? Does the issue warrant surfacing to a developer, or is it a trivially minor concern that adds noise without value?
q4: Is the issue introduced by this diff pre-existing or not? An issue present in code that the diff didn’t touch is not something this commit caused; thus, surfacing is misleading.
q5: Does the issue match its declared type? A poor_naming issue should describe a naming problem. A logic_error should describe incorrect logic. Type mismatches indicate the issue generator fired against the wrong category.
q6: Is the flagged code actually being removed by this diff? In unified diff format, lines prefixed with a minus sign are being deleted by the commit. Flagging code that the developer is already in the process of removing is not a useful signal.
The pass condition is precise: all of q1 through q5 must be true, and q6 must be false. A single failure filters the issue out.
Q6 is the most diff-specific of the six. It requires the judge to understand unified diff format conventions, specifically that a “-” prefix marks code being deleted, not code that will exist in the final result. This detail is not universally internalized by language models from training alone, as this series will cover in a later part.
The rubric is what makes vet reliable. It forces the judge model to evaluate each failure mode independently rather than making a general judgement call. It also makes the judge LLM replaceable: the rubric is explicit and testable, which means any candidate model can be evaluated against it before being trusted.
Why the Session Limit Became a Design Problem
Session limits are not a flaw in vet. They are the expected cost of delegating structured judgment to a cloud LLM at scale. The question they raise is whether that delegation is necessary.
The rubric is explicit. The task is narrow. The output is a structured verdict, not a generic description. These properties suggest that a judge does not necessarily require a frontier model. With Small Laguage Model (SLM) advancements in 2026, a smaller, locally-run model might handle q1 through q6 reliably, without session limits, without per-call token cost, and without sending large code diffs to an external API.
The practical appeal of a local judge is straightforward: consistent availability, and the ability to run on every commit without accumulating against a quota. The open question is whether “capable enough” is achievable at small scale. A local model that agrees with everything would solve the cost problem while undermining the quality guarantee entirely.
That question is worth investigating carefully. The rubric is explicit, the task is narrow, and the output is structured, which suggests the judgment step does not inherently require a frontier model. Whether a small local model can meet that bar is not obvious, and the answer turns on failure modes that are easy to overlook. Part 2 of this series [2] introduces the LLM judge concept in full, including the yes-bias, the failure mode that makes model selection more nuanced than it first appears. The story started with a tool that worked well enough to trust and a cost structure that made consistent use impractical. Whether a small model can preserve what made it worth trusting is the question this series sets out to answer.
References
[1] https://github.com/imbue-ai/vet
[2] Can You Trust an LLM Judge?
Great Article! I myself recently tried using LLM to judge RAG retrieval quality with some Rubric Questions, will look into yours to see how if I can improve.