Background Coding Agents with Strong Feedback Loops for Large-Scale Code Transformations

spotify · Media & Entertainment · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Reliable background coding-agent verification

Spotify needed background coding agents that could produce reliable PRs across thousands of software components without direct human supervision, while avoiding broken CI and functionally incorrect but passing changes.

Constraints:

• agents can fail to produce PRs
• agents can produce PRs that fail CI
• the most serious risk is PRs that pass CI but are functionally wrong
• components may lack tests, agents may go outside prompt scope, and build/test output can consume context
• the agent must remain limited and sandboxed for predictability and security

Architecture

Spotify designed verification loops around Honk. The agent edits code in a limited container, can call an abstract verifier tool through MCP, and automatically runs relevant verifiers before PR opening. Individual verifiers such as Maven are selected by component contents and parse build/test output into concise feedback. After deterministic verifiers pass, an LLM judge evaluates the proposed diff against the original prompt. Surrounding infrastructure handles Slack/user interactions, pushing code, prompt authoring, and future CI/CD outer loops, keeping the agent focused on one code-change task.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  prompt[Prompt] --> background-agent[Background agent]
  background-agent --> relevant-codebase[Relevant codebase]
  relevant-codebase --> verifier-selector[Verifier selector]
  verifier-selector --> independent-verifiers[Independent verifiers]
  background-agent --> mcp-verifier-tool[MCP verifier tool]
  mcp-verifier-tool --> independent-verifiers
  background-agent --> container-sandbox[Container sandbox]
  independent-verifiers --> pull-request-gate[Pull request gate]
  prompt --> llm-judge[LLM judge]
  llm-judge --> pull-request-gate
  background-agent --> pull-request-gate
  pull-request-gate --> surrounding-infrastructure[Surrounding infrastructure]
  surrounding-infrastructure --> future-ci-outer-loop[Future CI outer loop]

Components

Data flow

A prompt starts a background coding run. The sandboxed agent edits the codebase and invokes verifier tools; the MCP abstraction hides the underlying build/test details. Relevant verifiers run automatically before PR creation, failure messages guide correction, and the LLM judge evaluates whether the diff stays within prompt scope. Only after checks pass does the surrounding infrastructure open or manage the PR.

Why this architecture

Spotify reduces flexibility on purpose. Independent verifiers keep build/test noise out of the agent context, the LLM judge catches over-ambitious changes that deterministic tests miss, and the sandboxed agent does one task while infrastructure handles Slack, pushing, and prompts. This improves predictability and security for large-scale code transformations.

Models used: Honk · Claude Code · LLM judge

Scale

• The verification design targets thousands of software components.
  “On a high level, when we run agentic code changes across thousands of different [software components](https://backstage.”article, stated
• The LLM judge vetoes about a quarter of thousands of agent sessions.
  “However, we know from internal metrics that out of thousands of agent sessions, the judge vetoes about a quarter of them.”article, stated
• When the judge vetoes, the agent course-corrects half the time.
  “When that happens, the agent is able to course correct half the time. From empirical observations, we have seen that the most common trigger is the agent going outside the instructions outlined in the prompt.”article, stated

Alternatives considered (roads not taken)

Let the agent understand and run every build system directly

Rejected because: Parsing build/test specifics would waste context and add noisy decision-making.

Use deterministic verifiers only

Rejected because: Some agents made out-of-scope changes that tests/builds might not catch.

Give the agent broad system responsibilities

Rejected because: Pushing code, Slack interaction, and prompt authoring are intentionally handled outside the agent for predictability and security.

Patterns

eval-driven-development (verification)

Verification loops guide the agent and block PRs until builds/tests/judge checks pass.

llm-as-judge (verification)

An LLM evaluates the proposed diff against the original prompt after deterministic checks complete.

tool-use-orchestration (agent-runtime)

The agent calls abstract verifier tools through MCP while independent verifiers run the underlying checks.

guardrails-io-filtering (safety)

The agent has limited access in a highly sandboxed container and only handles code changes.

human-in-the-loop-gate (review)

When verifier failures block PRs, users receive an error instead of a half-broken change to merge.

Tradeoffs

Keep verifier details hidden from the agent

Chose MCP abstraction over independent verifiers over agent directly invoking each build/test system.

Rationale: The agent gets incremental feedback without spending context on noisy implementation details.

Reduce agent flexibility

Chose limited access and surrounding infrastructure for complex tasks over a broad autonomous agent with Slack, push, prompt-authoring, and system access.

Rationale: Less flexibility makes behavior more predictable and improves security.

Add an LLM judge despite lacking judge evals

Chose prompt/diff LLM evaluation after deterministic verifiers over waiting for a fully evaluated judge before deployment.

Rationale: Internal metrics showed the judge caught out-of-scope behavior and enabled course correction.

Gotchas (what bit them)

A passing CI result does not guarantee functional correctness.

Root cause: Agents can make changes that satisfy available tests while misunderstanding the prompt or behavior.

Resolution: Spotify adds an LLM judge over diff plus prompt to catch out-of-scope changes.

Build and test output can consume the agent's context.

Root cause: Different build systems and complex failure output are noisy and tedious to parse.

Resolution: Independent verifiers summarize only relevant errors and hide implementation detail behind MCP.

Agents can get creative beyond the prompt.

Root cause: They may refactor code or disable flaky tests instead of doing the requested narrow change.

Resolution: The LLM judge vetoes such diffs and can force course correction.

Verifier platform coverage limits adoption.

Root cause: Current verifiers run on Linux x86 while iOS/macOS and ARM64 systems need different hosts.

Resolution: Spotify plans to expand verifier infrastructure across hardware and operating systems.

Evaluation

Spotify evaluates reliability with deterministic verifiers, automatic pre-PR checks, LLM-judge veto and course-correction metrics, future CI/CD outer loops, and planned structured evals for prompts, architectures, and LLM providers.

Metrics: three failure modes: no PR, CI-failing PR, CI-passing but wrong PR · judge vetoes about a quarter of thousands of sessions · agent course-corrects half the time after judge veto · Linux x86 verifier coverage today · planned CI/CD outer loop and structured evals

Outcomes

• Verification loops guide background agents toward correct PRs before opening them.
• LLM judging catches over-ambitious changes that deterministic verifiers may miss.
• Spotify observed judge vetoes on about a quarter of thousands of sessions and course correction half the time.
• The architecture keeps the coding agent sandboxed and narrow while surrounding infrastructure handles complex workflow concerns.

The source reports thousands of agent sessions, judge veto/course-correction rates, and planned expansion beyond Linux x86 verifier environments.