The first article in this series introduced vet, a command-line tool that uses AI to generate code review issues and then validates each one against a six-question rubric before surfacing it to a developer [1]. That validation step represents a design pattern that appears across a growing range of AI systems. Understanding the pattern is what makes the model selection challenge in Parts 3 and 4 more than a benchmark exercise.

This article covers how the pattern works, why yes-bias is the most common and least obvious failure mode, and what properties to verify before trusting a candidate model.

Subscribe now

The Concept Defined

A standard language model interaction is generative: a prompt goes in and the model produces a response. An LLM judge works differently. It receives a structured input comprising two elements, the artifact to evaluate and the criteria by which to evaluate the artifact, and returns a verdict. The output is not speculation, it is a judgment.

The judge is not generating new content. It is discriminating: valid or not, pass or fail, accurate or inaccurate. The distinction matters because the properties that make a model good at generation (fluency, creativity, coherent long-form reasoning) are not the same properties that make it reliable at judgment (consistency, rubric adherence, appropriate skepticism).

The judge can evaluate outputs from a larger model, a smaller model, or the same model class. The size relationship between the generator and the judge is not the relevant variable. What matters is whether the judge’s verdicts are reliable. In vet’s case, the generated code issue is the artifact, the six-question rubric from Part 1 is the criteria, and the verdict is a pass or fail. The pattern is the same regardless of the implementation.

How It Works in Practice

A judge call has three core components: the artifact, the rubric, and the verdict format.

The artifact is whatever is being evaluated. In a code review context, that is a generated issue alongside the diff that prompted it. In other contexts it might be a customer support response, a generated summary, or a piece of information extracted from a document.

The rubric is the set of explicit criteria that define what “valid” means for this particular task. In vet’s case, the rubric includes questions such as: “Is the issue grounded in specific code, rather than in the absence of something?” and “Is the flagged code actually being removed by this diff?” Each criteria targets a concrete failure mode rather than leaving the model to form a general response. A rubric written this way is auditable; any case where the judge returns the wrong answer can be traced back to specific criteria that was misapplied or misunderstood. This pattern of decomposing evaluation into fine-grained per-criterion binary decisions is well-established in LLM judge research, appearing independently in systems such as G-Eval and HD-Eval [2].

The verdict is the judge’s structured output: a boolean per criterion, a score, or a final pass/fail. The key word is structured. A judge that returns free-form language about quality is difficult to integrate into an automated pipeline. A judge that returns a predictable, machine-readable verdict can be wired directly into the system. In a minimal implementation, the output might be as simple as {"valid": true, "criteria": [true, true, false, true, true, false]}, one boolean per rubric question. That structure is what allows a downstream pipeline to act on the verdict without parsing natural language.

It is worth noting that rubric design is a form of engineering in its own right. A vague rubric produces vague, unreliable judgments. A rubric that can be tested against a labeled dataset, where the correct verdict is known in advance, is considerably more trustworthy than one built on intuition about what the model will do.

Why It Matters: Scale and Consistency

The appeal of an LLM judge is scale. Human review of every AI-generated output is impractical when generation runs on every commit, every query, or every customer interaction. A judge enables quality filtering to run automatically at the same speed as generation, without requiring human attention on each individual case.

A judge also applies the rubric consistently. A human reviewer working under time pressure may weight criteria differently from one review to the next. The judge applies exactly the same rubric to every artifact it sees.

Two failure modes carry equal weight. A judge that lets invalid outputs through, producing false positives, erodes trust in the overall system. With enough noise, users stop paying attention to the results. A judge that filters valid outputs, producing false negatives, reduces coverage and undermines the system’s purpose. Both matter, and a well-calibrated judge minimizes both.

It is also worth stating the appropriate scope: a judge is a filter, not a replacement for human judgment at decision points that carry real consequences.

The Yes-Bias Problem

The most common judge failure mode is not inaccuracy on a specific criterion. It is a pervasive tendency to return “pass” regardless of what the model receives. This is called yes-bias, and understanding why it happens is important before evaluating any candidate model.

Language models are trained through a process called reinforcement learning from human feedback (RLHF), which optimizes for outputs that human raters find agreeable and helpful. Research on LLM evaluators identifies this as the root cause of what the field calls self-enhancement bias: a structural tendency to return favorable verdicts regardless of whether the input merits them [2]. A judge role requires the opposite: selective rejection, skepticism toward plausible-looking artifacts, willingness to return “fail” even when the input seems reasonable at first glance. These behaviors run contrary to what RLHF training reinforces.

A yes-biased judge can appear, superficially, to be performing well. If most artifacts it receives happen to be valid, a model that says “pass” to everything will show a high accuracy score. The problem becomes visible only when the dataset is balanced, with roughly equal valid and invalid cases. On a balanced dataset, a yes-biased model scores approximately 50%, identical to random guessing [2]. That is the test that exposes the problem before it enters production.

Yes-bias is also not detectable from general benchmark scores. A model that performs well on reasoning tasks or code generation can still be yes-biased in a judge role on a specific rubric. The only reliable way to check is to run the model against labeled scenarios where the correct answer is “fail,” and observe whether it returns “fail.” In practice, a yes-biased model looks deceptively functional at first: the response format is correct, there are no errors, and verdicts appear considered. The problem only surfaces when the expected answers are checked against the actual output.

The practical implication is direct: checking for yes-bias should be the first step when evaluating a judge candidate, before prompt tuning or system message work. A model that cannot return “fail” on clear negative cases cannot be tuned into a reliable judge. It is worth noting that the opposite failure exists as well: a model that rejects nearly everything regardless of the input. That variant scores equally poorly on a balanced dataset and fails in the same way, just in the other direction. The goal is calibration, not skepticism for its own sake.

What Makes a Reliable Judge

With yes-bias in mind, the properties that distinguish a reliable judge become concrete.

Determinism. At a fixed temperature setting, the same input should produce the same verdict on every call. A judge that returns different answers on identical scenarios introduces noise rather than removing it. Thus, benchmarking becomes unreliable and production behavior becomes unpredictable [2].

Rubric adherence. The model should evaluate against the stated criteria, not drift toward a general impression of quality. A judge that ignores a specific criterion in favor of a holistic “this seems fine” response is not applying the rubric [2].

Appropriate skepticism. The model should return “fail” when criteria are not met, even when the artifact looks plausible. This is the counterweight to yes-bias; a well-calibrated judge neither agrees with everything nor rejects everything.

Bias resistance beyond yes-bias. Yes-bias is the most common failure mode, but research on LLM judge systems identifies several others that are directly relevant to a code review context: length bias, where a model favors verbose issue descriptions regardless of validity; concreteness bias, where specific line numbers or technical terminology cause an issue to pass even when the underlying claim is wrong; and position bias, where verdicts shift based on where content appears within the prompt [2]. A rubric-based judge is partially protected by its structured criteria, but these biases are worth verifying on a sample of known-verdict cases before a candidate is promoted for use.

Context capacity. The judge receives the artifact plus the rubric and supporting context. On long inputs, a small context window risks truncation, which is particularly problematic because it is often silent: the model produces a verdict anyway, based on incomplete information. The practical countermeasures are straightforward; prefer models with a context window sized to the largest expected input, and keep the rubric concise so it does not consume a disproportionate share of available tokens.

All five of these properties are directly testable before deployment, and that is worth emphasizing. A small labeled evaluation dataset, even ten to fifteen representative scenarios with known correct verdicts, provides more reliable signal about judge performance than a general benchmark. Scores on standard capability benchmarks measure abilities that may not correspond to performance on a specific structured rubric in a specific domain. This is the practical entry point for evaluating a judge candidate: build the dataset first, run the candidate against it, and let the results guide every decision that follows.

Testing these properties fits well with my scenario. I have a structured rubric from vet to build a dataset with known answers. The judge candidate can then be validated against the artifact it generates, plus the rubric, and the dataset.

What Comes Next

The LLM judge is a pattern for systematic, scalable quality filtering of AI-generated output. Its reliability depends on a clear and testable rubric, determinism in the model’s responses, and resistance to known pitfalls. Those are the properties that separate a reliable judge from one that merely appears reliable. Understanding them concretely is what makes the model comparison ahead more than a ranking exercise; it shows which properties actually matter when the pattern has to work under real conditions.

Vet’s framework has these properties. The cost described in Part 1 [1], comes in the form of session token limits that make consistent coverage difficult on an active codebase. The question is whether a small, locally-run model can meet the same reliability bar without the per-call token spend.

Part 3 of this series examines the model landscape: what small language models are, why they are a reasonable fit for the judge role, and what properties to look for in the candidates selected for testing.

Stay tuned for the next article: SLMs vs LLMs: Choosing a Model for Judgment Tasks

References

[1] What is an LLM Judge?

[2] A Survey on LLM-as-a-Judge, arXiv:2411.15594v6

When writing code with AI and Spec Driven Development (SDD), I have found that there are still mistakes in spite of good specs. No matter how detailed or concise the spec, things were missed. When searching for a way to have “self healing” reviews to validate implementations, I came across many methods. One of the methods that seemed to work better than others was a utility from imbue-ai call “verify everything” (vet) [1]. This utility used python scripts and LLM as a judge to review diffs and score them to validate the spec was implemented correctly.

After a few weeks of regular use, vet caught real issues: logic errors that would have slipped through, silent failure patterns buried in error-handling code, and missed tests. Not every flagged issue was valid, but the signal-to-noise ratio was good enough that running vet on every commit felt worth the overhead

Then the session limits started appearing.

Vet’s judgment step, which decides whether a flagged issue is actually worth surfacing, runs against a cloud LLM by default (Anthropic or OpenAi). However, vet can use the existing agent if the cloud provider is not available. Using cloud or agent API connection for a judge comes with tradeoffs. The most common is session token limits, which often stopps work once hit.

Rather than accepting the limit as a fixed constraint, the more interesting question was whether a locally-run model could serve as the judge, eliminate the session limits, and maintain the quality that made vet useful in the first place. This article covers what vet is, how its LLM judge step works, and why that architecture makes a local replacement worth exploring. That investigation is what this series documents.

The vet Pipeline: From Diff to Developer

The flow from a code change to a surfaced issue passes through several stages. A developer commits code or opens a pull request. Vet’s issue identifiers analyze the diff and produce a set of candidate issues. Each candidate then passes through a judgment function that acts as a filter.

The judge receives four inputs: the unified diff, the commit message, the issue description, and a guide that defines what the issue type is meant to detect. It returns a verdict, pass or fail. Only issues that pass the verdict reach the developer.

The asymmetry here matters. False positives, invalid issues surfaced to the developer, erode trust over time. Enough noise and reviewers start dismissing vet’s output without reading it. False negatives, valid issues filtered out, reduce coverage and defeat the tool’s purpose. The system is calibrated to avoid false positives first, which means the judge is intentionally strict.

The token cost concentrates in the judgment step. Every candidate issue requires a separate call to the cloud LLM, with roughly 1,500 to 2,500 tokens of context per call. On an active codebase or large changes, this adds up quickly. The session limit is not a failure of the tool; it is the natural consequence of using a cloud API for every individual judgment call.

Subscribe now

The Six-Question Rubric

The judge does not form a general impression of whether an issue seems valid. It answers six specific yes-or-no questions, each targeting a distinct way an issue can be flawed.

q1: Is the issue grounded in specific code, rather than in the absence of something? An issue that says “there is no input validation here” is flagging missing code, not a defect in the code that exists. This question filters out absence-of-information issues, which cannot be verified from the diff alone.

q2: Does the issue avoid speculating about how code might be used, without evidence of that usage in the codebase? A concern like “this could be called with untrusted input” is speculation unless the code diff shows the actual function call. This question filters out issues that project hypothetical usage scenarios onto code the diff does not demonstrate being misused.

q3: Is the severity reasonable? Does the issue warrant surfacing to a developer, or is it a trivially minor concern that adds noise without value?

q4: Is the issue introduced by this diff pre-existing or not? An issue present in code that the diff didn’t touch is not something this commit caused; thus, surfacing is misleading.

q5: Does the issue match its declared type? A poor_naming issue should describe a naming problem. A logic_error should describe incorrect logic. Type mismatches indicate the issue generator fired against the wrong category.

q6: Is the flagged code actually being removed by this diff? In unified diff format, lines prefixed with a minus sign are being deleted by the commit. Flagging code that the developer is already in the process of removing is not a useful signal.

The pass condition is precise: all of q1 through q5 must be true, and q6 must be false. A single failure filters the issue out.

Q6 is the most diff-specific of the six. It requires the judge to understand unified diff format conventions, specifically that a “-” prefix marks code being deleted, not code that will exist in the final result. This detail is not universally internalized by language models from training alone, as this series will cover in a later part.

The rubric is what makes vet reliable. It forces the judge model to evaluate each failure mode independently rather than making a general judgement call. It also makes the judge LLM replaceable: the rubric is explicit and testable, which means any candidate model can be evaluated against it before being trusted.

Why the Session Limit Became a Design Problem

Session limits are not a flaw in vet. They are the expected cost of delegating structured judgment to a cloud LLM at scale. The question they raise is whether that delegation is necessary.

The rubric is explicit. The task is narrow. The output is a structured verdict, not a generic description. These properties suggest that a judge does not necessarily require a frontier model. With Small Laguage Model (SLM) advancements in 2026, a smaller, locally-run model might handle q1 through q6 reliably, without session limits, without per-call token cost, and without sending large code diffs to an external API.

The practical appeal of a local judge is straightforward: consistent availability, and the ability to run on every commit without accumulating against a quota. The open question is whether “capable enough” is achievable at small scale. A local model that agrees with everything would solve the cost problem while undermining the quality guarantee entirely.

That question is worth investigating carefully. The rubric is explicit, the task is narrow, and the output is structured, which suggests the judgment step does not inherently require a frontier model. Whether a small local model can meet that bar is not obvious, and the answer turns on failure modes that are easy to overlook. Part 2 of this series [2] introduces the LLM judge concept in full, including the yes-bias, the failure mode that makes model selection more nuanced than it first appears. The story started with a tool that worked well enough to trust and a cost structure that made consistent use impractical. Whether a small model can preserve what made it worth trusting is the question this series sets out to answer.

References

[1] https://github.com/imbue-ai/vet

[2] Can You Trust an LLM Judge?

The intuition behind using multi-agents or sub-agents with Kiro is straightforward: more agents working in parallel means faster output. It is a reasonable assumption, and Kiro makes it easy to act on. As of version 0.8, Kiro supports synchronous sub-agents across the IDE, CLI, and ACP clients. Kiro agents can also invoke other agents, giving you access to two common multi-agent topologies that fit the majority of needs [1]. However, picking the right topology pattern is less about capability and more about understanding where each pattern breaks down. A 2026 research paper makes the failure mode precise, and the number is worth knowing before you split a task across multiple agents [2].

Two Patterns, Two Trade-offs

Kiro includes the Maker-Evaluator sub-agent topology by default, with Hub-and-Spoke easily accessible [1]. Both are well-suited to specific tasks, but neither is a general-purpose upgrade over a single agent.

Maker-Evaluator is a two-node loop: one agent produces output, a second evaluates it. When the same agent plays both roles, the pattern is called reflection. The main agent spawns a sub-agent and is blocked until the sub-agent completes. The main agent then decides whether to re-invoke based on the evaluation result.

This pattern fits iterative quality work: code review, coder-plus-tester automated turns, document generation with a writer-editor pair, and security auditing. Its token cost does not compound across iterations the way Hub-and-Spoke does across spokes, and failure recovery is simpler because the loop state is centralized.

Two failure modes are worth calling out before you wire up this pattern.

1. Runaway loop. Without a termination condition, the loop will not stop on its own. The best mitigation is a hard iteration cap of three turns.

2. Quality degradation. Quality gains flatten after roughly three turns, which Sartori’s research explains directly [2]. Using an adversarial evaluator with explicit, written rejection criteria and a defined standard rather than an implicit one will help maintain quality.

Hub-and-Spoke is a star topology where one central hub agent decomposes a task, delegates subtasks to up to four spoke agents, and aggregates their results. Spokes do not communicate with each other in this topology. The hub holds the full picture; spokes operate in isolation.

This pattern works well where tasks are genuinely independent: a planner agent decomposing requirements, a coder agent implementing them, a tester agent validating the output. It also suits orchestrator-plus-specialist setups where each spoke owns a distinct domain.

The costs for this topology are real and compound quickly. The hub is a single point of failure: if it loses context or produces a malformed delegation, the entire workflow stalls. Kiro credit costs also accumulate across every spoke independently and count toward your total usage in parallel. Those sub-agent spokes are not free. File writes are a concern when multiple spokes write to the same files, causing collisions.

Three mitigation strategies are worth building in from the start.

1. Accommodate parallel writes. Git Worktrees isolate parallel write paths so spokes do not overwrite each other’s changes, at the cost of disk space.

2. Optimize inter-agent communication. Terse hub-to-spoke messages, sometimes called caveman mode, can reduce token overhead.

3. Use persistent external memory. A “Document As You Go” pattern of structured session logs gives agents access to shared context that would otherwise be lost between invocations.

The Multi-Agent Coordination Gap

Think of two developers building the same feature in separate rooms with no way to talk to each other. Even with a perfect written spec, they will make small, incompatible decisions: one uses a list, the other uses a dictionary. Merging their work is harder than if one person had built the whole thing.

This is precisely what Sartori 2026 [2] set out to measure, and the results may surprise you. With a full specification, a single agent succeeded 89% of the time. Two agents working in parallel succeeded only 58%. With almost no specification at all, the single agent dropped to 56%, and the two-agent split dropped to 25%. The gap never closed, regardless of how detailed the spec was.

A better spec helps both approaches equally. It does not close the gap between them. The 25-to-39 percentage point penalty is structural: it comes from the agents not sharing decisions, not from the quality of the instructions.

This is why the Maker-Evaluator loop has a ceiling around three turns. Every additional iteration is two agents coordinating again, each time incurring that same overhead. More turns do not compound quality; they compound cost.

The Decision

Think of pattern selection as a per-task trade-off rather than a project-wide default.

Hub-and-Spoke is the right choice when tasks are genuinely independent, when parallelism matters more than accuracy on any individual task, and when the hub can synthesize partial outputs without needing agents to share state. If those conditions are not met, you absorb the coordination penalty without the parallelism benefit.

Maker-Evaluator is the right choice when iterative quality improvement is the goal, when the iteration count is bounded at three turns, and when the evaluator is independent of the implementor. An evaluator agent that shares context with the implementor absorbs the full coordination gap without providing independent verification.

A single agent remains the right choice when the task requires shared context across steps, when the codebase contains implicit invariants that agents cannot reliably infer, or when correctness is the primary constraint and the coordination penalty is unacceptable.

Three practical anchors apply regardless of which pattern you choose. Kiro allows custom agent definitions that specify model, tools, and context per sub-agent; use this to constrain scope rather than expand it. The intent document, or steering file, is the only persistent state between stateless agent sessions; without it, context resets on every invocation. For Maker-Evaluator specifically, write the termination condition before you wire the loop, because the model will not determine when enough is enough on its own.

Multi-agent is not a free upgrade. Hub-and-Spoke trades accuracy per task for parallelism. Maker-Evaluator trades compounding latency and a hard quality ceiling for iterative improvement. Sartori’s data makes the cost explicit: 25 to 39 percentage points of coordination overhead, present at every specification level, recoverable only by concentrating context in a single agent [2]. Knowing when not to split is the decision that makes the other two patterns worthwhile.

Ready? Build your first Kiro Agent in 15 Minutes

References

[1] J. Wasowski, After Analyzing 17 Multi-Agent Topologies - 7 Anti-Patterns That will Burn Your Budget, 2026 https://medium.com/gitconnected/after-analyzing-17-multi-agent-topologies-7-anti-patterns-that-will-burn-your-budget-28cc6909621c

[2] C. Sartori, The Specification Gap: Coordination Failure Under Partial Knowledge in Code Agents, 2026 https://doi.org/10.48550/arXiv.2603.24284

When an AI agent like Kiro executes commands, modifies files, or runs scripts, it does so with whatever permissions your local environment provides. For workflows involving sensitive data, production credentials, or complex automation, this introduces meaningful risk. Sandboxing defines explicit boundaries around what Kiro can and cannot do, reducing that risk without sacrificing productivity.

This guide is written for developers using Kiro on macOS. It covers what sandboxing is, why it matters, and three practical methods for applying it to your workflow.

Table of Contents

• What Is Sandboxing?

• Why Sandboxing Is Needed

• Can Hooks Replace a Sandbox?

• How to Sandbox Kiro on macOS

• Feature Comparison

• Advantages of Sandboxing

• Limitations of Sandboxing

• Conclusion

What Is Sandboxing?

Sandboxing means running an AI agent, or the commands it issues, inside a restricted and isolated environment. The agent retains the ability to perform actions, but only within carefully controlled boundaries.

Consider the analogy of a workshop with a locked tool cabinet. A craftsperson can work freely within the workshop, using whatever tools are available, but cannot access anything locked away or work outside the designated space. Kiro operates in a similar fashion: it can execute commands, modify files, and run scripts, but only within the limits you define.

Why Sandboxing Is Needed

Without defined boundaries, an AI agent operates with the same access rights as the developer running it. In practice, this means it could accidentally:

• delete important files

• expose API keys or credentials stored in shell profiles

• damage the operating system

• modify files outside the intended project scope

• access private data stored elsewhere on disk

Consider a command as simple as:

rm -rf /

Without restrictions, this would destroy an entire macOS system. Sandboxing prevents this class of accident by restricting what the agent is permitted to touch before any command executes.

To see this in practice: suppose Kiro needs to run python app.py. With sandboxing, execution is constrained to /home/user/project. It cannot reach system folders, modify directories outside the project, or read sensitive files elsewhere. The operation completes normally, but the damage from any mistake is contained.

Can Hooks Replace a Sandbox?

Kiro supports hooks, which are rules that inspect and can block actions before they are executed. A common question is whether hooks provide sufficient security on their own. The short answer is: they help, but they are not a substitute.

A useful way to frame the distinction is:

• Hooks are specific checks

• A sandbox is the boundary itself

Hooks work by inspecting commands, checking for dangerous patterns, and allowing or blocking behavior accordingly. They are a valuable layer of defense. However, hooks are implemented in software and carry the same vulnerabilities as any software system. If a hook fails to match the exact command invocation used, the command proceeds. A sandbox restricts access at the environment level regardless; the operating system enforces the boundary, not the hook.

Hooks and sandboxing are complementary, not competing. Hooks provide targeted checks; sandboxes provide the hard boundary that catches what hooks miss.

How to Sandbox Kiro CLI on macOS

There are three primary approaches, each suited to a different level of isolation and workflow complexity.

1. Use a Virtual Machine (UTM)

A virtual machine (VM) provides the most complete form of isolation available. It creates an entirely separate macOS environment, with its own kernel, filesystem, and user profile, running as a guest on your physical machine. Kiro, installed inside the VM, has no visibility into your host machine’s files, credentials, or shell configuration.

For macOS on Apple Silicon, UTM is the recommended choice. It is free, and well-suited to running macOS guest environments.

Setup steps:

1. Download and install UTM from the official site (mac.getutm.app) or from the Mac App Store.

2. Open UTM and select Virtualize from the start screen.

3. Choose macOS 12+ as the operating system.

4. Import or download the macOS installer. If UTM offers the option to continue without selecting an IPSW file, it will use the installer on your boot partition.

5. Set RAM and CPU limits appropriate to your host machine.

6. Set a disk size for the virtual environment.

7. Review and save the configuration on the Summary screen. The VM will appear in the left sidebar.

8. Launch the VM using the play button. Initial setup will take several minutes to complete.

Once the guest environment is running and kiro-cli is installed inside it, the agent operates in complete isolation. Your host machine’s Documents folder, Desktop, credentials, and shell profiles remain invisible to it.

2. Use a Container (Docker)

Docker containers offer a lighter-weight alternative to a full VM. They share the host macOS kernel but isolate the filesystem, processes, and network from the host environment. If a script runs incorrectly inside the container, only the container is affected; the host machine remains unchanged.

Setup steps:

1. Create a Dockerfile:

FROM node:lts-slim
RUN apt-get update && apt-get install -y git curl python3 build-essential
WORKDIR /workspace
RUN curl -fsSL https://cli.kiro.dev/install | bash

2. Build and tag the image:

docker build -f Dockerfile -t kiro-sandbox .

3. Start the container, mounting only the specific project directory Kiro should access:

docker run -it --rm \
  -v $(pwd):/workspace \
  kiro-sandbox bash

3. Use Application-Level Isolation (SRT)

For developers who prefer to work directly on their local machine without a VM or container, application-level isolation provides a practical middle ground. Anthropic’s Sandbox Runtime (srt) wraps kiro-cli in a declarative sandbox, enforcing filesystem and network boundaries using Apple’s native Seatbelt security framework on macOS.

It’s worth noting that srt is an experimental tool. Its configuration and API may evolve over time.

Setup steps:

1. Install the Sandbox Runtime via npm:

npm install -g @anthropic-ai/sandbox-runtime

2. Create the configuration file at ~/.srt-settings.json. The paths listed under allowWrite must include the application’s data directory, or the tool will not start correctly:

{
  "allowPty": true,
  "enableWeakerNestedSandbox": true,
  "enableWeakerNetworkIsolation": true,
  "network": {
    "allowedDomains": [
      "*.kiro.dev",
      "*.amazonaws.com",
      "*.awsapps.com",
      "*.aws.dev"
    ]
  },
  "filesystem": {
    "allowWrite": [
      "./workspace",
      "~/Library/Application Support/kiro-cli/",
      "~/.kiro"
    ]
  }
}

3. Launch kiro-cli inside the sandbox:

srt kiro-cli chat

On macOS, srt hooks into Apple’s Seatbelt framework to enforce these boundaries at the kernel level. Any attempt by Kiro to write outside the allowWrite list or reach an unlisted domain is blocked before it executes.

Feature Comparison

The three approaches differ not just in setup complexity, but in where the isolation boundary sits in the computing stack. The table below compares them across eight technical dimensions.

Advantages of Sandboxing

• Blast Radius Containment: If something goes wrong, damage is confined to the sandbox; the host machine is unaffected and the sandbox can simply be discarded.

• Secure Execution of Untrusted Code: Safely test unfamiliar scripts or third-party tools without auditing every line in advance.

• Protection Against Zero-Day Exploits: Restricting what an application is permitted to do limits potential damage from vulnerabilities that have not yet been disclosed or patched.

• Clean, Reproducible Environments: Each instance starts from a known state, free of stale configuration or leftover artifacts from previous sessions.

Limitations of Sandboxing

• Performance and Resource Overhead: Isolation boundaries require computational resources. Virtual machines in particular introduce meaningful CPU, RAM, and startup time costs that may affect developer experience.

• Sandbox Escape Vulnerabilities: No isolation mechanism is perfect. A flaw in the sandbox implementation can allow a process to break containment. External data sources, such as repositories containing hidden Unicode payloads, represent a less obvious escape vector.

• Context Blindness and Friction: The sandbox may block Kiro’s access to local files or internal tools it legitimately needs. Some configuration is necessary to restore that access without reopening the boundaries the sandbox is meant to enforce.

• Sandbox-Aware Behavior: Some malicious software detects when it is running inside a test environment and suppresses its harmful behavior, only revealing itself once it reaches an unrestricted host.

• Trusted-Channel Data Poisoning: Allowlisting domains like github.com or google.com grants network access but cannot sanitize the content returned. A prompt injection payload in a repository file, or adversarially crafted search results, can manipulate Kiro’s behavior from within the sandbox. The boundary controls what the agent can reach, not what it reads or how it interprets that content.

• No Defense Against Agentic Attack Vectors: Sandboxing operates at the system and network level and does not address the behavioral attack surface of an AI agent. The OWASP Top 10 for LLM Applications identifies risks sandboxing cannot mitigate: goal hijacking, tool misuse, identity abuse, unexpected code execution, and context poisoning. These attacks target model reasoning, not the host OS, and pass through sandbox boundaries undetected.

Conclusion

Sandboxing controls what Kiro can access at the system and network level; it does not govern how the agent reasons about or responds to the content it retrieves. For macOS developers, the three approaches covered here represent a progression from maximum isolation to minimum friction.

• A virtual machine is the right choice when the risk profile demands the strongest possible guarantee, such as working with sensitive data or long-running agents.

• A container is a sensible default for most development work, offering solid isolation with familiar tooling and low overhead.

• Application-level isolation via srt is well-suited to developers who need to stay close to their local environment while still enforcing meaningful boundaries around what Kiro can access.

Kiro CLI ships with a default agent (kiro_default), but custom agents let you go further. A custom agent is a named configuration that gives an LLM a specific role, a defined set of tools, and context loaded automatically at startup. Rather than repeating the same prompt setup every session, a custom agent captures it once and makes it instantly available to you and your team.

This tutorial will walk through building a code-reviewer agent to review code with controlled tool access and automatically load a project README on startup. By the end, you will have a working local agent file you can activate, swap to, and commit to version control.

What you will build: a code-reviewer agent that reads files and runs shell commands without prompting, loads your project README automatically, and greets you when activated.

Time: ~15 minutes

Prerequisites

• Kiro CLI installed and authenticated (version >= 2.1)

• An active chat session (kiro-cli chat)

• A project directory with a README.md under version control (git)

Step 1 — Understand where agents live

Kiro looks for agents in two places:

Location

• .kiro/agents/ in your project — Scope: only available for the project

• ~/.kiro/agents/ — Scope: available everywhere

When both locations contain an agent with the same name, the local version takes precedence. This makes local agents a good choice when you want behavior tailored to a specific project, while global agents are better suited for general-purpose assistants you reach for everywhere.

For this tutorial, a local agent keeps things contained to your project.

To begin, create the agents directory in the project:

mkdir -p .kiro/agents

With the directory in place, the next step is creating the configuration file.

Step 2 — Create the agent file

Create .kiro/agents/code-reviewer.json with the following content:

{
  "name": "code-reviewer",
  "description": "Reviews code changes. Reads files and runs git commands without prompting.",
  "prompt": "You are a thorough code reviewer. Focus on correctness, clarity, and security. Be concise.",
  "tools": ["read", "shell"],
  "allowedTools": ["read", "shell"],
  "resources": [
    "file://README.md"
  ],
  "welcomeMessage": "Ready to review. Share a file path or paste a diff."
}

What each field does:

• tools — declares what the agent can use

• allowedTools — declares what runs without a permission prompt

• resources — files loaded into context when the agent starts

• welcomeMessage — shown when you switch to this agent

Save the file. Kiro detects new agent files automatically, no restart is required for the agent to appear in the list

Note on config changes: adding a new agent file takes effect immediately. Changes to an existing agent’s configuration, however, take effect the next time you activate the agent (via /agent swap). A running session does not reload mid-conversation.

Step 3 — Activate the agent

Start a chat session:

kiro-cli chat

Inside the session, swap to your new agent:

 /agent

Select code-reviewer from the list. You will see:

✔ Choose one of the following agents · code-reviewer
Ready to review. Share a file path or paste a diff.
code-reviewer · auto

Your README.md is already loaded in context. To confirm, ask the agent something about it:

code-reviewer · auto
What does this project do?

The agent answers using the README content. No file-reading prompt appears because read is in allowedTools and runs silently by design.

Step 4 — Test tool permissions

To see the permission boundary in action, ask the agent to inspect recent changes:

code-reviewer · auto 
What files have changed?

The agent runs git status without prompting, because shell is pre-approved within allowedTools. Now try something outside its approved list:

code-reviewer · auto 
Write a summary to NOTES.md

Kiro will prompt you for permission before writing, because write is not listed in allowedTools. This is the security boundary working as intended.

Step 5 — Restrict write access with toolsSettings

You decide the agent should be able to write, but only to a reviews/ directory. Exit Kiro and update the config to add write capability to both tools and allowedTools:

{
  "name": "code-reviewer",
  "description": "Reviews code changes. Reads files and runs git commands without prompting.",
  "prompt": "You are a thorough code reviewer. Focus on correctness, clarity, and security. Be concise.",
  "tools": ["read", "write", "shell"],
  "allowedTools": ["read", "shell", "write"],
  "toolsSettings": {
    "write": {
      "allowedPaths": ["reviews/**"]
    }
  },
  "resources": [
    "file://README.md"
  ],
  "welcomeMessage": "Ready to review. Share a file path or paste a diff."
}

Create the directory:

mkdir reviews

Start a new session with kiro-cli chat (config changes take effect on next chat activation), and swap to the code-review agent with commands from Step 3:

# activate new session
kiro-cli chat
# activate the code-review agent
/agent

Now ask the agent to write a review:

code-reviewer · auto 
Review project files and save findings to reviews/main-review.md

The agent writes to reviews/main-review.md without prompting. An attempt to write anywhere else will still require confirmation.

Troubleshooting

Agent does not appear in the /agent list

Check that the file is valid JSON — a missing comma or bracket will silently prevent the agent from loading. A JSON linter or jq . .kiro/agents/code-reviewer.json can surface syntax errors quickly.

Resource file not found warning

Kiro resolves file:// paths relative to the project root. If README is in a subdirectory, update the path to match: file://docs/REAME.md

Config changes not taking effect

Changes to an existing agent require re-activation. Run /agent to change agents and then swap back to reload the config in the current session.

Conclusion

• Agent files live in .kiro/agents/ (local) or ~/.kiro/agents/ (global)

• tools declares availability; allowedTools removes the permission prompt

• toolsSettings constrains what allowed tools can touch (e.g., allowedPaths for write operations)

• resources pre-load files into context at startup

Next steps

• Move the prompt to a separate file: "prompt": "file://./prompts/code-reviewer.md" for easier editing

• Commit .kiro/agents/code-reviewer.json to version control so teammates get the same agent automatically

• Read the official configuration reference for all available fields

The conversation around AI-driven software development has never been louder. From Amazon’s opinionated AI-Driven SDLC1 to the growing body of work around Spec-Driven Development2 , the industry is moving quickly to position AI as the definitive solution to an age-old challenge: how do we build software faster, more reliably, and at greater scale?

These frameworks offer genuine value. The productivity gains for teams that adopt them are real, and the examples are compelling. However, the discourse consistently makes the same mistake: it scopes “the SDLC” to mean the build phase and little else.

The true software development lifecycle is far broader. It encompasses Ideation, Architecture, Planning, Build, Operations, Fixes, and Retirement, a complete arc from the first spark of a product idea to the deliberate decommissioning of a system. When we evaluate AI’s role through this wider lens, a more honest and more complicated picture emerges. AI is not a replacement for the SDLC process. It is, at its best, a powerful accelerator but only when its integration into the lifecycle is deliberately and carefully architected.

Why Small Teams Don’t Prove the Case

It is worth acknowledging what AI-first development gets right. For solopreneurs, small startups, and lean product teams, an AI-first SDLC can be genuinely transformative. The context window of a modern AI system is sufficient to hold the full scope of a small codebase. One or two engineers can move with a speed that would have been impossible a few years ago. The gains are real.

The challenge is that these successes are being used to justify adoption at an entirely different scale, large enterprises with hundreds or thousands of engineers, hundreds of services, and decades of accumulated architectural decisions. The properties that make AI effective for a small team do not transfer cleanly to this environment.

At enterprise scale, the assumptions break down. A single AI agent cannot hold the full context of a distributed system spanning dozens of teams and hundreds of services. The clean feedback loop between a developer and an AI assistant becomes a tangled web of dependencies, competing priorities, and organizational constraints. Understanding why requires examining the most fundamental problem with AI at scale: non-determinism.

The Non-Determinism Problem

Non-determinism is not a quirk of current AI systems that will eventually be engineered away. It is an inherent property of the probabilistic models that power them. For small-scale development tasks (writing a function, generating a test suite, summarizing documentation), this is largely acceptable. The cost of variation is low, and a human reviewer catches the drift.

At enterprise scale, the cost of non-determinism compounds rapidly.

Consider a large organization running hundreds of microservices across multiple teams. Each service represents a distinct bounded context, typically owned by a small team. Features and epics orchestrate work across these boundaries at a higher level. If an AI planning agent is responsible for generating specifications across multiple epics simultaneously, it must do so consistently (not just within a single session), but across repeated invocations, different teams, and changing business context.

This is where Spec-Driven Development begins to strain. Ask an AI agent to define the acceptance criteria for a given feature, and it will produce a reasonable answer. Ask it again tomorrow, with slightly different phrasing, and the answer will shift. At small scale, this is manageable. Across hundreds of services and dozens of teams, this drift accumulates into inconsistency that is difficult to detect and expensive to correct.

The deeper issue is accountability. Human developers navigate ambiguity through judgment, context, and professional accountability. When a decision leads to a poor outcome, there is a person who made that call and can learn from it. When an AI agent makes the same decision, a decision from a system that is non-deterministic by design, accountability becomes diffuse. Who owns the output? Who is responsible when acceptance criteria shift between sprints and the resulting system fails to meet business needs? These are not rhetorical questions. They are organizational challenges that must be answered before AI can be safely integrated at scale.

Architecture Consistency at Scale

The non-determinism problem is most consequential when it touches architectural decisions. For large organizations, technical architecture is not a creative exercise, it is a discipline. Architecture principles, patterns, and guidance must be applied consistently across the landscape if that landscape is to remain manageable over time.

Consider what happens when architecture is delegated to an AI without constraints. Asked to design a new service, the AI might select a microservices approach. Asked again for a different service with similar requirements, it might favor a modular monolith. Asked a third time, it might propose an event-driven architecture. While each choice may be individually defensible, collectively they produce a fragmented landscape where every service is a unique artifact, each with its own operational patterns, its own failure modes, and its own runbooks.

This fragmentation also has direct operational and financial consequences. Operations teams cannot apply generalized expertise across services that each behave differently. Incident response becomes slower because runbooks cannot be standardized. Cloud costs become difficult to manage because cloud resource selections (Lambda versus EC2 versus ECS versus EKS), vary by service rather than following a consistent decision framework. FinOps programs, which depend on predictable patterns to optimize spend, are undermined by this inconsistency.

The solution is not to exclude AI from architectural decisions, but to constrain the space in which it operates. Guardrails, prescriptive pattern sets, and architectural governance frameworks give AI agents a bounded set of valid choices. Within that bounded space, AI can accelerate architectural work significantly. Outside it, the long-term costs outweigh the short-term gains.

Operations: The Hidden Cost of Upstream Decisions

Operational complexity does not emerge at deployment time. It is designed in (or more accurately, it is neglected) at the planning and architecture phases. Every decision made upstream about how a service is structured, what data it produces, and how it communicates with its neighbors has a direct consequence for how it will operate in production.

This is the hidden cost that AI-SDLC frameworks consistently underestimate.

For AI agents to participate meaningfully in operations (like detecting anomalies, diagnosing failures, triggering remediations) they require rich, consistent observability signals. Logs must be structured and semantically meaningful. Metrics must cover the right indicators. Distributed traces must propagate correctly across service boundaries. These are not implementation details that can be added after the fact. They must be part of the initial planning phase; rather than discovered as gaps during the first production incident.

This introduces a critical architectural requirement: the feedback loop between operational agents and planning agents must be explicitly designed. When an operational agent encounters a failure it cannot diagnose because the necessary signals are missing, that information must flow back to the planning and architecture layers. The planning agent that generated the original specification must be capable of receiving and incorporating this feedback. Without this loop, the system learns nothing from production, and the same observability gaps are reproduced in every subsequent service.

Furthermore, operational agents cannot be generic. A service built on a Lambda-based event-driven pattern has fundamentally different failure modes than a service built on a long-running container. Effective operational AI requires specialization, agents that understand the specific patterns they are operating in, not agents that reason from first principles about every incident. This in turn, reinforces the argument for architectural consistency: a landscape with fewer distinct patterns requires fewer specialized agents and produces more predictable operational outcomes.

The Phases Nobody Talks About

The build phase receives the majority of attention in AI-SDLC. This is understandable because it is where the most visible productivity gains occur, and it is the phase most responsive to automation. However, a lifecycle that begins at planning and ends at deployment is not a lifecycle ... it is a fragment.

Ideation is where software begins. Before a line of code is written, before an architecture is selected, business context must be translated into product requirements. This is an activity that involves stakeholder negotiation, market understanding, strategic judgment, and organizational politics. AI can assist with this phase: synthesizing research, generating initial requirement drafts, identifying gaps in specifications. However, the judgment about what to build and why remains a human responsibility. AI-SDLC frameworks that begin at the planning phase are implicitly assuming that ideation has already been resolved, which is rarely true in practice.

Fixes represent a continuous parallel track to feature development. Bug reports, production incidents, and security vulnerabilities do not pause while the planning agent generates the next sprint’s epics. AI agents that operate in the fixes track face a different set of constraints than those operating in the feature track: they must reason from incomplete information, work against time pressure, and frequently operate on legacy code that predates any AI involvement. Integrating fixes into an AI-SDLC requires explicit tooling for incident context ingestion, prioritization logic, and safe rollback mechanisms, none of which are addressed in current frameworks.

Retirement and migration may be the most neglected phase of all. Every dependency within a service has a lifecycle of its own. Programming languages release new versions, and libraries reach end-of-life. When a core technology in a service’s stack loses community support or vendor maintenance, the cost of inaction compounds over time with security exposure, incompatibility with adjacent services, and eventual forced migrations under time pressure.

A complete AI-SDLC must account for this. It requires a dedicated monitoring capability with an agent or set of agents whose responsibility is tracking the dependency health of every service in the portfolio. When a dependency approaches end-of-life, the system should surface that signal to the planning layer before it becomes a crisis ... not after. This is not an unexpected requirement; rather, it is the operational reality of maintaining software at scale, and it is almost entirely absent from current AI-SDLC thinking.

What a Real Enterprise AI-SDLC Requires

Having examined the gaps, it is possible to sketch out an AI-SDLC that actually works at enterprise scale. It is not a single agent, a single framework, or a single vendor’s platform. It is a system of systems and a set of specialized, constrained agents operating within a governance structure that preserves human accountability at the strategic layer.

The foundational principle is a clear division of responsibility. Human orchestrators own the “what” and the “why.” They define business strategy, set architectural principles, establish governance guardrails, and make the calls that carry accountability. AI agents own the “how”, by executing within the boundaries that human orchestrators define, accelerating the tedious and dependency-heavy work of implementation, and surfacing information that humans need to make better decisions.

This division only functions if the following components are in place:

Guardrails and constrained pattern sets. Architectural AI agents must operate within a defined set of approved patterns. The set should be small enough to maintain consistency and large enough to cover legitimate variation. Deviations from approved patterns should require human approval, not AI discretion.

Observability-first specifications. Planning agents must generate specifications that include explicit observability requirements, like what logs the service must produce, what metrics it must expose, what traces it must propagate. These requirements are not optional and must be validated before a service is considered complete.

Explicit feedback loops. Operational agents must have a defined channel to communicate signal gaps and failure patterns back to planning and architecture agents. This loop closes the connection between what was designed and what is actually happening in production.

Dependency monitoring. A dedicated agent or capability must track the health and lifecycle status of every dependency across the portfolio, surfacing EOL risks to the planning layer on a continuous basis.

Human accountability checkpoints. Non-deterministic outputs in architectural decisions, acceptance criteria, and migration plans must pass through human review before they are committed to. AI generates the options; humans make the call.

Together, these components address the non-determinism problem not by eliminating it, but by containing it. AI operates freely within bounded, reversible, low-stakes decisions. Human judgment intervenes at the high-stakes, high-consequence points where accountability matters.

Conclusion: The Real Revolution

The most important shift in enterprise software development is not the adoption of AI. It is the recognition that AI adoption requires architectural thinking of the same rigor and care as any other major system integration.

Organizations that treat AI-SDLC as a tool swap by replacing the old process steps with new AI-driven equivalents, will encounter the compounding costs described throughout this article: Fragmented architectures, Inconsistent observability, Untracked dependencies, and Accountability gaps that surface at the worst possible moments.

Organizations that succeed will be those that design the integration deliberately: defining the boundaries within which AI operates, building the feedback loops that keep the system honest, and preserving human judgment at the points where it matters most.

The question facing enterprise software leaders is not “how do we adopt AI in our SDLC?” It is a more precise and more demanding question: “how do we architect a system in which AI and human judgment each do what they do best, across the full lifecycle of every service we operate?”

That question does not have a simple answer. It requires the same systems thinking, strategic clarity, and organizational discipline that have always separated organizations that manage complexity well from those that are managed by it. AI does not change that requirement. Rather, it makes meeting that requirement more achievable than it has ever been.

2

https://www.thoughtworks.com/radar/techniques/spec-driven-development