Benchmarking SLM as a Judge for AI Code Review
Part 4 of the Local AI Code Review series
This journey started after hitting Anthropic’s session limits one too many times, and asking the question: can a local model actually do the judge’s job? There was only one way to find out.
Three locally-run small language models were evaluated against vet’s six-question rubric using a curated dataset of twelve scenarios. Each model was given two rounds of iteration: a baseline run, then one system-message tuning pass. The first sign that something was seriously wrong with one of the candidates was not a failed test case. It was the realization (mid-run), that the model had not returned a single “fail” across any scenario. That is when the yes-bias problem stopped being a theoretical concern and became a disqualifying result.
The full context for what follows is in the earlier parts of this series: vet’s pipeline and q1–q6 rubric are covered in Part 1 [1], the LLM judge pattern and yes-bias failure mode in Part 2 [2], and the three candidate models and their properties in Part 3 [3]. This article covers what happened when all of it was put to the test.
The Benchmark Setup
The dataset consisted of twelve curated scenarios, referred to as “goldens”, spanning all five of vet’s issue categories: fails_silently, insecure_code, logic_error, runtime_error_risk, and poor_naming. Each scenario provided a commit message, a unified diff, an issue description, and an issue type code. The expected output was a single word: “pass” or “fail.”
The evaluation metric was an exact match, the model’s verdict against the known correct answer. No secondary LLM scorer was involved; no external API key was required. The evaluation could run entirely offline.
The dataset was balanced across expected outcomes, with roughly equal pass and fail cases. This balance was intentional: as Part 2 described, a yes-biased model will score approximately 50% on a balanced dataset with all-pass responses, which makes the problem immediately visible rather than hiding behind a skewed distribution. Using balanced positive and negative labels to surface yes-bias before it enters real use reflects established practice in LLM judge meta-evaluation research [4].
Each model was allowed one baseline run and one round of system-message refinement. Injecting guidance into the user turn was ruled out after prior experiments showed it caused unexpected regressions on unrelated criteria.
Model Results
Apple Foundation Model (apfel): The Agreeable Model
After the first full run, apfel had returned “pass” on every single scenario; including the six where the expected answer was “fail.” Not a borderline result; a complete absence of discrimination across the entire dataset. Round 1 score: 50% on a balanced dataset, which is precisely what random agreement looks like.
Across multiple runs, the best observed result was approximately 58%, or seven out of twelve. The root cause was consistent: the model returned “pass” on roughly 95% of inputs regardless of scenario content. Rubric criteria that required skepticism, specifically q2 (concrete evidence), q4 (introduced by this diff), q5 (type scope match), and q6 (removed code detection), were all treated as true by default.
Several tuning attempts were made. Narrowing the scope guidance for q5 helped marginally. Explicit q6 instruction for removed-code detection was unreliable across runs. Broad instructions to “be stricter” caused regressions on valid cases where the model then began filtering issues it should have passed.
The disqualifying factor, beyond the accuracy ceiling, was non-determinism at temperature zero. Two runs on identical scenarios produced different verdicts. A judge must be consistent; this model was not. The combination of yes-bias, an accuracy ceiling of approximately 58%, and non-determinism made apfel unsuitable for this role regardless of further tuning effort.
A model that agrees with everything is not a judge. It is noise in a different form.
Qwen3:8b: The Thinking Model Trap
Qwen3:8b opened with a more promising baseline: 75%, or nine out of twelve, in Round 1. After system-message tuning in Round 2, it reached 91.67%, or eleven out of twelve. The integration, however, produced an unexpected obstacle.
Qwen3 has an extended thinking mode that activates by default when called through the standard OpenAI-compatible API. In this mode, the model allocates a large portion of its output token budget to internal reasoning before producing its final response. On judge calls, this exhausted the available token budget and produced truncated or unusable responses. The solution was to route calls through the Ollama-native endpoint with extended thinking explicitly disabled. With that in place, the model behaved predictably and deterministically at temperature zero. Using an “instruct” version should achieve similar results.
Round 1’s three failures were all false positives: the judge passed issues that should have been filtered.
The first involved code that both logs an exception and re-raises it. Qwen3 accepted this as a fails_silently issue, which it is not — the exception surfaces to the caller. This was a q5 miss; the issue was categorized under the wrong type. The second and third failures involved speculative risks: a theoretical concurrent-modification scenario and a theoretical null-reference scenario, neither of which had concrete supporting evidence in the diff. Both were q2 misses; the model accepted possibilities as facts.
Round 2’s system message targeted both failure modes directly: explicit guidance that q2 requires observable evidence in the submitted code, and that issue descriptions must match the scope of the declared type. Two of the three failures resolved. The third, the null-reference scenario, remains a borderline case. The diff contained a field access on an object that could theoretically be null; fixing the verdict risked regressing a closely related scenario that is legitimately a valid issue. The residual failure was accepted, and the model closed at 91.67%.
The lesson is narrow but important: the model is capable; the default thinking mode is the hazard. Disable it explicitly, or route through the native endpoint.
Granite4.1:8b: Clean API, Clean Results
Granite4.1:8b also opened at 75% in Round 1, but its failure modes were different in character from Qwen3’s. Where Qwen3’s errors were false positives, two of Granite’s three Round 1 failures were false negatives: the model over-filtered valid issues it should have passed.
The first failure involved q6. One scenario flagged code that appeared on a line being removed from the diff, where the relevant diff line began with a minus sign. Granite set q6 incorrectly, treating the flagged code as present in the resulting state when it was actually being deleted. This is a subtle convention: in unified diff format, a - prefix marks a line being removed, and flagging code that is already going away is not a valid issue. The model had not internalized this convention from training.
The second and third failures were two valid issues — a logic_error and a runtime_error_risk — where Granite applied unnecessary skepticism to unambiguous code. Both were clear, concrete defects with direct evidence in the diff; the model filtered them when it should have passed them.
Round 2 addressed all three. An explicit q6 instruction was added to the system message: if the flagged line starts with a minus sign in the unified diff, the code is being removed, and q6 should be set accordingly. A calibration instruction was added alongside it: trust unambiguous, concrete code evidence without requiring additional context. Both changes were specific to the failure modes observed; neither affected unrelated criteria.
Round 2 result: 12 out of 12, with no residual borderline cases. Integration throughout used the standard OpenAI-compatible endpoint with no workarounds, and the 131,072-token context window provided headroom for realistic judge payload.
Five Lessons for Building Your Own Judge
The results across three models and two rounds of iteration produced findings that generalize beyond vet and beyond these specific models.
1. Check for yes-bias before tuning. Run the baseline against a balanced dataset first. A model at 50% on balanced labels with all-pass responses is disqualified, regardless of its general benchmark scores. No amount of system-message work reliably fixes a fundamental tendency toward agreement. Discovering this at the baseline stage costs one evaluation run; discovering it after weeks of tuning costs considerably more.
2. System messages are safer than prompt injection. Injecting clarification into the user turn caused unexpected side effects on unrelated rubric criteria in prior experiments. System messages scope guidance cleanly. One instruction per criterion is the right granularity; broader instructions tend to produce broader, less predictable effects. Research on LLM judge prompt design independently confirms that evaluation criteria and output format constraints belong in the system prompt rather than the user turn [4].
3. False negatives are as harmful as false positives. Over-filtering valid issues reduces the system’s coverage and its value to developers. Granite’s Round 1 false negatives were resolved by a single calibration instruction, without introducing regressions elsewhere. The fix was specific to the observed failure mode, not a broad adjustment. As Part 1 described, both failure directions carry real cost.
4. Diff format is semantic. The + and - prefixes in unified diffs carry meaning the model must understand to evaluate q6 correctly. This convention is not universally internalized from training. An explicit sentence in the system message resolves the gap reliably; without it, a capable model will still make consistent errors on removed-code scenarios.
5. Plan for context headroom. At 1,500 to 2,500 tokens per judge call, a 4,096-token ceiling is uncomfortably close on larger pull requests or when project context is included in the prompt. The session-limit problem that motivated this entire investigation was a cost problem; running into truncation would trade one reliability problem for another.
The Decision
Granite4.1:8b was selected as the judge model for my local setup.
It is the only candidate to reach 100% accuracy on the benchmark dataset out of the three. It is deterministic at temperature zero, uses a standard API without workarounds, and has the largest context window of the three candidates. The system message developed during the evaluation carries directly into the judge configuration with no modifications required.
The dataset is twelve scenarios; accuracy on the full distribution of real pull requests will differ, and the benchmark should be treated as a floor rather than a ceiling. The plan is to expand the dataset as edge cases surface, and to re-run the benchmark against any model update before using a new version.
It is worth noting what this result illustrates about general benchmarks. Survey-scale evaluations of comparable 7-8B open-source models on general human-alignment tasks place accuracy in the mid-50% range [4]. Granite reached 100% on this task-specific rubric after two rounds of iteration. The gap is not explained by model capability alone; it reflects rubric narrowness. When the task is well-defined and the evaluation dataset is labeled against specific criteria, task-specific performance systematically exceeds what general benchmark scores predict. That is the argument for building a labeled dataset before making any model selection decision.
So What: Beyond vet
The pattern applied here is transferable. Any AI system that generates candidates requiring quality filtering is a candidate for an LLM judge layer. The evaluation discipline, not the specific models or rubric, is the portable contribution: a small labeled dataset, explicit rubric criteria, and two rounds of system-message iteration is sufficient to identify whether a local model is viable for the role.
Local small language models are viable as judges for narrow, well-defined tasks. That viability is not inherent to the model class benchmarks. Fit for purpose depends on checking the right properties, being willing to discard candidates that fail the baseline, and being precise about the failure modes the rubric is meant to catch.
The question that opened Part 1 was whether eliminating session limits was possible without sacrificing the judgment quality that made vet useful. The answer, with the right model and the right evaluation process, is yes.
Read the full series from the beginning
References
[1] https://www.aaronp.net/p/what-is-an-llm-judge
[2] https://www.aaronp.net/p/can-you-trust-an-llm-judge
[3] https://www.aaronp.net/p/slms-vs-llms-choosing-a-judging-model
[4] A Survey on LLM-as-a-Judge, arXiv:2411.15594v6, Oct. 2025.
