Building Reliable Background Coding Agents with Verification Loops

spotify · Media & Entertainment · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Background coding agents for large-scale software maintenance

Spotify needed background coding agents to make automated code changes across thousands of software components without direct human supervision while avoiding PRs that fail CI or, worse, pass CI while being functionally wrong.

Constraints:

• The agent runs across thousands of heterogeneous software components.
• A failed-to-produce PR is tolerable, but broken or incorrect PRs consume engineering time and erode trust.
• Passing CI is not sufficient when agents make out-of-scope functional changes.
• Different build systems and test outputs can consume the agent's context window.
• The coding agent should remain limited and predictable rather than managing prompting, Slack, pushing code, and infrastructure tasks itself.
• Verifier support must eventually expand beyond Linux x86 to macOS and ARM64 for broader adoption.

Architecture

Spotify's Honk background coding agent is intentionally narrow: it receives a prompt, sees the relevant codebase, edits files, and invokes verifiers as tools. Surrounding infrastructure handles prompt authoring, Slack interactions, pushing code, and other orchestration. The core reliability mechanism is a verification loop exposed through MCP: independent verifiers activate automatically based on component contents, such as a Maven verifier when a root pom.xml exists, and return concise feedback so the agent can iterate before opening a PR. All relevant verifiers also run before PR creation through a Claude Code stop hook. After deterministic formatting/build/test verifiers pass, an LLM judge compares the proposed diff with the original prompt to veto out-of-scope changes. The agent runs in a highly sandboxed container with limited permissions, few binaries, and little access to surrounding systems.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  surrounding-infrastructure[Surrounding infrastructure] --> background-coding-agent[Background coding agent]
  background-coding-agent --> relevant-codebase[Relevant codebase]
  relevant-codebase --> verifier-activation[Verifier activation]
  verifier-activation --> independent-verifiers[Independent verifiers]
  background-coding-agent --> mcp-verifier-tool[MCP verifier tool]
  mcp-verifier-tool --> independent-verifiers
  independent-verifiers --> stop-hook-gate[Stop hook gate]
  stop-hook-gate --> pull-request-flow[Pull request flow]
  independent-verifiers --> llm-judge[LLM judge]
  llm-judge --> pull-request-flow
  surrounding-infrastructure --> pull-request-flow
  background-coding-agent --> sandboxed-runtime[Sandboxed runtime]

Components

Data flow

Surrounding infrastructure sends a scoped code-change prompt to the background coding agent. The agent edits the relevant codebase and calls an MCP verifier tool without needing to know individual verifier internals. Component contents activate independent verifiers, which run formatting, build, and test checks and return concise feedback. Before a PR opens, all relevant verifiers run again through a stop hook; failures block the PR and surface an error. If deterministic verifiers pass, the LLM judge receives the proposed diff and original prompt, vetoing changes that drift outside the request. Only changes that survive those loops reach the pull request flow.

Why this architecture

Spotify designed for predictability because functionally incorrect PRs that pass CI are hard to catch across thousands of components and can break production. The verification abstraction gives the agent feedback without forcing it to understand build systems or parse noisy test output. The LLM judge exists because deterministic tests cannot detect all out-of-scope refactors or disabled flaky tests. The agent is kept narrow and sandboxed so surrounding infrastructure, not the model, owns higher-risk operations.

Models used: Underlying LLM for the background coding agent · LLM judge for diff-versus-prompt evaluation · Claude Code stop hook integration

Scale

• Spotify runs agentic code changes across thousands of software components.
  “when we run agentic code changes across thousands of different software components”article, stated
• The article identifies three primary failure modes for background coding agents.
  “we worry about three primary failure modes.”article, stated
• The judge vetoes about one quarter of thousands of agent sessions.
  “out of thousands of agent sessions, the judge vetoes about a quarter of them.”article, stated
• When the judge vetoes a change, the agent course-corrects about half the time.
  “When that happens, the agent is able to course correct half the time.”article, stated
• The current verifier platform covers Linux x86 but not all needed macOS or ARM64 environments.
  “Our current verifiers only run on Linux x86 and while that serves our backend and web infrastructure well”article, stated

Alternatives considered (roads not taken)

Expose each verifier implementation directly to the agent

Rejected because: Spotify hides verifier details behind an MCP abstraction so the agent only knows it can or must call verification tools.

Rely only on deterministic formatting, build, and test verifiers

Rejected because: Deterministic checks can pass even when an agent goes outside prompt scope by refactoring or disabling flaky tests.

Let the coding agent own all surrounding workflow tasks

Rejected because: Spotify deliberately keeps pushing code, Slack interaction, and prompt authoring outside the agent to reduce flexibility and improve predictability.

Open PRs before the full verifier loop runs

Rejected because: A stop hook blocks PR creation when any relevant verifier fails, reducing review noise and half-broken PRs.

Patterns

eval-driven-development (quality)

Spotify routes every proposed change through deterministic verifiers and plans structured evals for system prompts, architectures, and providers.

llm-as-judge (verification)

An LLM judge evaluates proposed diffs against the original prompt after deterministic verifiers pass, vetoing out-of-scope changes.

tool-use-orchestration (architecture)

Verifiers are exposed as a single MCP tool abstraction, while individual checks activate automatically from component contents.

context-compaction (context)

Verifiers parse noisy build and test output, returning only relevant errors or short success messages so the agent's context window is not consumed by raw logs.

guardrails-io-filtering (safety)

The stop hook and judge gate prevent failed verification or out-of-scope changes from becoming PRs.

safety-policy-enforcement (security)

The agent is sandboxed with limited permissions, few binaries, and little access to surrounding systems.

Tradeoffs

Keep the background agent narrowly scoped

Chose An agent that takes a prompt, edits code, and calls verifiers over An agent that also manages pushing code, Slack interaction, and prompt authoring.

Rationale: Reduced flexibility improves predictability and security.

Hide verifier internals behind MCP

Chose A unified verifier tool abstraction over Teaching the agent each individual verifier and build system.

Rationale: The abstraction reduces context burden and lets component contents choose the right verifier automatically.

Run deterministic verifiers before the LLM judge

Chose Judge only after formatting, build, and test checks complete over Asking the judge to evaluate syntactically broken or untested code.

Rationale: The judge can focus on prompt adherence and out-of-scope behavior once lower-level correctness has been checked.

Block PR creation on verifier failure

Chose Stop-hook gate before PR opening over Creating PRs and letting humans discover verifier failures later.

Rationale: Blocking failed changes avoids noisy half-broken PRs and protects reviewer time.

Gotchas (what bit them)

A PR can pass CI while still being functionally wrong.

Root cause: Agents can make creative, out-of-scope changes that existing tests do not catch, especially when target components have weak test coverage.

Resolution: Add an LLM judge that compares the diff to the original prompt after deterministic verifiers pass.

Reviewing nonsensical or half-broken PRs becomes expensive at scale.

Root cause: Agents that fail CI or produce unclear changes force engineers to decide whether to salvage or abandon partially completed work.

Resolution: Run verifiers before PR creation and block PRs when verification fails.

Build and test output can overwhelm the agent context window.

Root cause: Different build systems and complex test outputs contain noisy details the agent does not need to reason over directly.

Resolution: Verifier implementations parse output and return concise, actionable success or failure messages.

Current verifier infrastructure does not cover every runtime environment Spotify needs.

Root cause: Existing verifiers only run on Linux x86, while iOS and some backend systems require macOS or ARM64.

Resolution: Expand verifier infrastructure to more hardware and operating systems.

The judge itself lacks robust evals so far.

Root cause: Spotify has internal metrics for judge veto behavior but has not yet invested in a structured judge-evaluation harness.

Resolution: Add robust evals for prompts, architectures, and provider benchmarks.

Evaluation

Spotify evaluates the background agent through independent formatting/build/test verifiers, an LLM judge that compares diffs against prompts, internal metrics on judge vetoes and course correction, and planned structured evals for prompts, architectures, and LLM providers.

Metrics: PR produced or failed to produce · CI pass or failure · Functional correctness despite passing CI · Verifier pass/fail by component · Judge veto rate · Agent course-correction rate after a veto · Future prompt, architecture, and provider benchmark results

Outcomes

• The LLM judge vetoes about a quarter of thousands of agent sessions.
• After judge vetoes, the agent course-corrects half the time.
• Spotify reports agents can solve increasingly complex tasks with high reliability when verifiers and the judge guide them.
• Without feedback loops, the agents often produce code that does not work.
• The article does not report merge rate, production incident reduction, or developer-hours saved.

The system operates across thousands of software components and thousands of agent sessions. The article quantifies judge behavior at roughly one-quarter vetoes and half course-correction after veto, and notes current verifier support is Linux x86 with macOS and ARM64 still needed.