Context Engineering and Tool Design for Background Coding Agents at Scale

spotify · Media & Entertainment · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Large-scale software maintenance with autonomous coding agents

Spotify needed background coding agents to produce reliable, mergeable pull requests across real-world codebases and thousands of repositories. The challenge moved from merely running an agent loop to designing prompts and tools that kept the agent predictable, testable, and context-efficient at migration scale.

Constraints:

The agent must operate across thousands of repositories with different build configurations.
Prompts must be reusable enough for migrations but precise enough to avoid impossible tasks.
Tool access must support verification without introducing too many unpredictable dimensions.
Context windows can be overwhelmed by broad file selection or multi-change tasks.
The team lacks fully structured evaluation for prompt/model quality, so operational safeguards matter.

Architecture

Spotify's architecture combines Fleet Management with Claude Code/Anthropic Agent SDK, migration prompts that specify end states, and a constrained tool surface: a verify MCP tool for formatters, linters, and tests; a standardized Git tool; and a strict Bash allowlist. Users or upstream workflow agents condense context into prompts before the coding agent runs.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  fleet-management-system["Fleet Management system"] --> migration-prompt["Migration prompt"]
  migration-prompt --> claude-code-agent["Claude Code agent"]
  workflow-context-agents["Workflow context agents"] --> migration-prompt
  claude-code-agent --> verify-tool["Verify tool"]
  verify-tool --> build-and-test-systems["Build and test systems"]
  claude-code-agent --> git-tool["Git tool"]
  git-tool --> pull-request-pipeline["Pull request pipeline"]
  claude-code-agent --> bash-allowlist["Bash allowlist"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Fleet Management system`	Spotify system extended by background coding agents to run large-scale code migrations.	`Claude Code agent`, `Migration prompt`
`Migration prompt`	Static, version-controllable instruction bundle that defines preconditions, examples, desired end state, and tests.	`Claude Code agent`, `Workflow context agents`
`Claude Code agent`	Top-performing background coding agent that interprets goals, edits code, manages tasks, and uses constrained tools.	`Migration prompt`, `Verify tool`, `Git tool`, `Bash allowlist`, `Pull request pipeline`
`Verify tool`	MCP tool that runs formatters, linters, tests, and build-system checks while reducing noisy output.	`Claude Code agent`, `Build and test systems`
`Git tool`	Standardized Git interface with selective subcommand exposure and protected operations.	`Claude Code agent`, `Pull request pipeline`
`Bash allowlist`	Limited shell command surface for safe utility commands such as ripgrep.	`Claude Code agent`
`Build and test systems`	Repository-specific systems invoked through the verify tool to decide whether the migration is complete.	`Verify tool`
`Pull request pipeline`	Production path where successful background-agent changes are opened and merged.	`Claude Code agent`, `Git tool`, `Fleet Management system`
`Workflow context agents`	Separate agents or workflows that can produce condensed prompts and relevant context for the coding agent.	`Migration prompt`

Data flow

A migration prompt plus scoped context enters the background coding agent. The agent edits code, runs verification through a dedicated tool that summarizes build/test output, uses constrained Git operations, and produces PRs when tests pass or retry/turn limits are reached.

“The agent takes a few turns editing files based on the prompt, incorporating feedback from the build system with each turn.”article, stated

“because our agent needs to operate on thousands of existing repositories with very different build configurations. This also allows us to reduce noise in the tool output by summarizing logs into something more digestible for the agent.”article, stated

Why this architecture

Spotify moved toward Claude Code plus constrained tools because the homegrown loop required rigid instructions, struggled with multi-step edits, and forced users to hand-pick context. Larger static prompts and a small tool surface made agent behavior easier to reason about and test at migration scale.

“Our homegrown agentic loop had key challenges — it required overly rigid instructions and struggled with complex, multistep edits. We needed an agent that could interpret high-level goals, manage tasks dynamically”article, stated

“we prefer to have larger static prompts, which are easier to reason about. You can version-control the prompts, write tests, and evaluate their performance.”article, stated

Models used: Claude Code · Anthropic Agent SDK · Goose · Aider

Scale

Spotify applied the Claude Code approach to about 50 migrations.
“As of today, Claude Code is our top-performing agent, which we’ve applied for about 50 migrations and the majority of the background agent PRs merged into production.”article, stated
The system targets thousands of repositories.
“writing good prompts — and verifying that the agent did the right thing — becomes significantly more difficult if you want to apply a change over thousands of repos.”article, stated
The article credits the system with thousands of PRs across hundreds of repositories.
“Spotify built a large scale system that used Claude Agent SDK to merge thousands of PRs across hundreds of repositories.”article, stated
The earlier custom loop had explicit execution limits of 10 turns and three retries.
“The task is complete once the tests are passed or limits are exceeded (10 turns per session, three session retries total).”article, stated

Alternatives considered (roads not taken)

Open-source agents such as Goose and Aider

Rejected because: They were impressive on simple exploration but did not reliably produce mergeable PRs at Spotify's migration scale.

“But we quickly ran into issues when we tried to scale these agents to our migration use case. It was difficult to get them to reliably produce mergeable PRs.”article, stated

Spotify's own rigid agentic loop

Rejected because: It worked for small changes but became unruly for multi-file changes, context selection, and longer operations.

“This worked well for small changes, for example, editing a deployment manifest, swapping out a config flag, or changing a single line of code. But for anything beyond that, it quickly became unruly.”article, stated

Dynamic MCP tools for fetching more context during the task

Rejected because: More tools can handle ambiguity, but they make the agent less testable and introduce more dimensions of unpredictability.

“While this does make the agent capable of tackling more complex and ambiguous tasks, this also makes it less testable and predictable. The more tools you have, the more dimensions of unpredictability you introduce.”article, stated

Patterns

`tool-use-orchestration` (orchestration)

The coding agent is powerful but intentionally constrained to a verify MCP tool, standardized Git access, and a Bash allowlist rather than a broad tool ecosystem.

“We keep our background coding agent very limited in terms of tools and hooks so it can focus on generating the right code change from a prompt.”article, stated

`eval-driven-development` (evaluation)

Prompts define desired end states through tests, and the verify tool runs formatters, linters, and tests so the agent can iterate against objective feedback.

““Make this code better” is not a good prompt. The agent needs a verifiable goal so it can iterate on a solution as it goes.”article, stated

`prompt-versioning` (prompt-operations)

Spotify favors larger static migration prompts because they can be version-controlled, tested, and evaluated like an artifact.

“we prefer to have larger static prompts, which are easier to reason about. You can version-control the prompts, write tests, and evaluate their performance.”article, stated

`guardrails-io-filtering` (governance)

Git operations are selectively exposed and standardized so the agent can commit safely without pushing or mutating origin unexpectedly.

“This allows us to selectively expose git subcommands (e.g., never push or change origin) and standardize others (e.g., setting the committer and using standardized commit message formats).”article, stated

Tradeoffs

Agent runtime

Chose Claude Code with Anthropic Agent SDK over open-source agents and a homegrown LLM loop.

Rationale: Claude Code handled high-level goals, tasks, and longer operations better than the previous alternatives for Spotify's production migration use case.

“To solve this, we turned to Claude Code, which allowed us to use more-natural task-oriented prompts, and we benefited from its built-in ability to manage Todo lists and spawn subagents efficiently.”article, stated

Prompt and context strategy

Chose larger static, version-controlled prompts over simpler prompts plus more dynamic MCP context tools.

Rationale: Static prompts were easier to reason about and test, while extra dynamic tools increased unpredictability.

“An alternative approach is to start with a simpler prompt but connect to Model Context Protocol (MCP) tools that allow the agent to dynamically fetch more context as it works on the problem.”article, stated

“While this does make the agent capable of tackling more complex and ambiguous tasks, this also makes it less testable and predictable.”article, stated

Migration prompt scope

Chose one change at a time over combining several related changes into one elaborate prompt.

Rationale: Combining changes increases the chance of context exhaustion or partial delivery.

“It can be convenient for users to combine several related changes into one elaborate prompt, but it’s more likely to get the agent in trouble because it will exhaust its context window or it will deliver a partial result.”article, stated

Gotchas (what bit them)

Context selection by users did not scale.

Root cause: The homegrown loop required users to choose exact files with git-grep, making broad searches overwhelm context and narrow searches starve the agent.

Resolution: Move to Claude Code's stronger task management and context handling, while condensing relevant context into prompts or workflow-agent outputs.

“Picking a too-broad search pattern — like git grep -l '*.java' — could easily overwhelm the context window of the LLM. Conversely, picking a too-limited search pattern would not allow the agent to solve the problem”article, stated

Multi-file edits exhausted simple agent loops.

Root cause: Cascading changes could run out of turns or lose the original task as the context window filled.

Resolution: Use an agent that can manage high-level goals, task lists, and subagents more dynamically.

“For this use case, our loop often ran out of turns. It also tended to get lost when it filled up its context window, forgetting the original task after a few turns.”article, stated

Too many tools made the agent less predictable.

Root cause: Dynamic MCP context tools introduce capability but add extra dimensions of behavior that are hard to test.

Resolution: Constrain the tool surface and make prompts carry the relevant context up front.

“The more tools you have, the more dimensions of unpredictability you introduce.”article, stated

“We instead ask users to condense relevant context into the prompt up front, either by including all relevant information in the prompt before handing it off to the agent”article, stated

Evaluation

Operational verification combines prompt-defined end states, test/formatter/linter execution through the verify tool, and production PR merge outcomes; the post admits structured prompt/model evaluation is still immature.

Metrics: tests passed · formatters and linters passed · PRs merged into production · about 50 migrations

“we are still flying mostly by intuition. Our prompts evolve by trial and error. We don’t yet have structured ways to evaluate which prompts or models perform best.”article, stated

Outcomes

Claude Code became Spotify's top-performing background coding agent.
Spotify applied it to about 50 migrations and merged most background-agent PRs into production.
The architecture produced thousands of PRs across hundreds of repositories, according to the cited Anthropic comment.

The case is explicitly enterprise-scale: thousands of repositories, hundreds of repositories with merged PRs, and roughly 50 migrations.

“As of today, Claude Code is our top-performing agent, which we’ve applied for about 50 migrations and the majority of the background agent PRs merged into production.”article, stated

“Spotify built a large scale system that used Claude Agent SDK to merge thousands of PRs across hundreds of repositories.”article, stated