SLMs vs LLMs: Choosing a Judging Model
Part 3 of the Local AI Code Review series
The latest frontier LLMs are remarkably capable. They are also often more than what a narrow, structured task requires. A judge model evaluating a six-question rubric is not generating responses to general questions, reasoning through ambiguous ethics, or producing creative output. It is making a sequence of decisions against well-defined criteria.
For that kind of task, the question worth asking is whether a small, locally-run model can do the job reliably, and in doing so, eliminate the session-limit problem that started this journey. Part 1 of this series introduced vet and the token cost friction at its judge step [1]. Part 2 explained what makes an LLM judge reliable [2]. This article examines the model landscape for filling that role locally.
When this evaluation began, the instinct was to look for the most capable small model available. The results shifted that to a different question: most capable at what? This article covers the four properties that actually predict suitability for the judge role, and profiles three specific models evaluated in Part 4, including one that failed the baseline test entirely.
What Makes a Model “Small”
A small language model (SLM), typically refers to a model in the range of one to ten billion parameters that is designed to run on consumer or developer hardware without a data center GPU [3]. The term is relative and evolving; what counted as a large model two years ago may now be described as small by practitioners running inference on a laptop in 2026.
The key practical distinction from a deployment perspective is not the parameter count but what it enables: an SLM runs locally, without a cloud API call, and without transmitting data to a third party. That changes the cost structure, the privacy posture, and the operational dependency profile of anything that uses it.
The capability gap between SLMs and frontier models is real on complex reasoning and broad knowledge tasks. For narrow, well-defined tasks with structured output requirements, that gap is smaller than intuition suggests. The question is always whether the task is narrow enough that the model’s capability ceiling does not become the limiting factor. My hypothesis is that a judge model evaluating a rubric with six explicit criteria, against a diff and a brief issue description, is a reasonable candidate for a narrow-enough task.
Why Local Execution Matters for a Judge
Running a judge locally offers practical advantages that are worth stating.
No per-call cost. A cloud API charges per token. At 1,500 to 2,500 tokens per judge call, consistent usage across an active codebase accumulates quickly. A local model has no marginal cost per call once the hardware is running; the only cost is the infrastructure already in place for development.
Low and predictable latency. A local model responds in seconds on developer hardware; cloud round-trip time can vary depending on load, context window size, and network conditions. For a tool integrated into a software development workflow, latency consistency matters as much as average speed.
Version-pinned reproducibility. Cloud judges are subject to silent model updates; the model behind an endpoint could change without notice, altering evaluation behavior in ways that are difficult to detect or debug. A local model runs a specific, fixed version whose output is stable across time. For a judge wired into a software development workflow, consistent behavior across runs matters as much as correct behavior on any single run [4].
The tradeoff is real: local deployment requires hardware capable of running the model and an operator responsible for managing model versions and behavior. Neither of these is a trivial commitment, but for a team or person accustomed running development infrastructure, the overhead is manageable.
Properties That Matter for the Judge Role
Not all small models are equivalent for judgment tasks. Research on LLM judge systems documents that smaller open-source models tend to produce greater evaluation inconsistency and more frequent misalignment with human judgments compared to frontier models [4]. The properties that follow are specifically the ones where small models most commonly fail; they are also different from what general benchmark rankings measure.
Determinism. At temperature zero, the same input should produce the same output on every call. A non-deterministic judge introduces inconsistency that makes both debugging and benchmarking unreliable. This is not guaranteed by every model even at zero temperature; it is worth verifying experimentally on at least a small set of identical inputs before committing to a candidate. Setting temperature to zero as the baseline for judge evaluation is independently established as standard practice in LLM judge research [4].
Yes-bias resistance. As Part 2 described, a model that agrees with everything is not a useful judge. Checking for yes-bias is a prerequisite, not an afterthought. A direct test: send the model a scenario that clearly fails one of the rubric criteria and observe whether it returns “fail.” A model that still returns “pass” has a fundamental bias that prompt tuning is unlikely to correct reliably.
Context window capacity. The judge call includes the diff, commit message, issue description, issue type guide, and system message. On larger pull requests, this payload can reach 2,500 tokens or more. A model with a 4,096-token context ceiling has limited headroom; a ceiling of 40,000 tokens or more provides comfortable capacity for real-world usage.
API surface simplicity. Some models have non-standard API requirements: extra parameters that must be excluded, special endpoint paths, or modes that interfere with structured output. Each workaround adds a failure surface. A model that works with a standard API and no special handling reduces the integration risk.
Rubric adherence over general reasoning. A model can score well on standard benchmarks and still fail to follow a structured rubric consistently. Task-specific evaluation against known scenarios is more predictive than general capability scores for this use case.
Three Candidate Models
The three models evaluated in Part 4 illustrate the range of what is currently available for local judge deployment in Q2 of 2026. Each looked promising on paper. The properties that mattered most only became apparent when each was run against the actual rubric. That gap between profile and performance is exactly why the evaluation in Part 4 exists.
Apple Foundation Model (apfel) runs natively on Apple silicon hardware, and is bult-in to macOS 26+ [5]. This makes apfel attractive for developers already working on Apple silicon. Its context window is 4,096 tokens, the smallest of the three candidates, which leaves limited headroom on longer diffs or when additional project context is included in the prompt. Availability is also currently limited to specific hardware configurations.
Qwen3:8b is an 8.19 billion parameter model from Alibaba’s Qwen team, available via Ollama [6]. Its context window is 40,960 tokens, which is comfortable for typical judge payloads. It has one notable integration concern: an extended thinking mode that activates by default when using the standard OpenAI-compatible API. When active, this mode can exhaust the output token budget and produce incomplete responses. The model is deterministic at temperature zero when the thinking mode is disabled, and it requires explicit suppression to behave predictably in a structured output role.
Granite4.1:8b is an 8.79 billion parameter model from IBM Research, also available via Ollama [7]. Its context window is 131,072 tokens, the largest of the three candidates. It uses the standard OpenAI-compatible API with no known parameter quirks or required workarounds, and it is deterministic at temperature zero. The large context window means it can handle large pull requests and extended project context without truncation risk.
What to Evaluate Before Committing
The evaluation checklist that emerges from Parts 2 and 3 is straightforward: test for yes-bias, verify determinism on repeated identical inputs, confirm the context window fits the realistic payload size, and confirm the API surface requires no special handling.
General benchmark scores are a starting point for shortlisting, not a finishing line. They measure capabilities that may not correspond to performance on a specific structured rubric. A small labeled dataset, as few as ten to fifteen scenarios with known correct verdicts, provides substantially more useful signal for this purpose.
Part 4 of this series applies exactly this evaluation to the three models above, using vet’s q1–q6 rubric and a curated twelve-scenario dataset. The results are more instructive, and more varied, than the model profiles above might suggest.
It is worth restating the opening premise before the benchmark: for a task as narrow as evaluating a six-question rubric against a code diff, a model does not need to be frontier-sized. It needs to be deterministic, resistant to yes-bias, and capable of fitting the full context. Those are the properties Part 4 actually tests.
Stay tuned for the next article: Benchmarking SLMs as a Judge for AI Code Review
References
[2] Can You Trust an LLM Judge?
[3] What are Small Language Models?
[4] A Survey on LLM-as-a-Judge, arXiv:2411.15594v6, Oct. 2025.
[5] Apple’s Foundation Models framework unlocks new app experiences powered by Apple Intelligence
[6] Qwen3:8b
[7] Granite4.1:8b
