Background Coding Agents for Large-Scale Software Maintenance and Migrations

spotify · Media & Entertainment · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Large-scale software maintenance and migrations

Spotify's existing Fleet Management platform handled simple fleet-wide code transformations well, but complex migrations still required scarce experts and too much manual work.

Constraints:

• complex transformations through AST manipulation or regular expressions required specialized expertise
• a Maven dependency updater had grown to more than 20,000 lines to handle corner cases
• background coding agents can be slow, unpredictable, costly, and safety-sensitive
• the system needed to fit existing targeting, review, PR, and production-merge workflows
• hundreds of developers and many repositories needed a preconfigured tool rather than bespoke agent setup

Architecture

Spotify evolved Fleet Management by replacing deterministic transformation declarations with prompts executed by a background coding agent while preserving the surrounding fleet infrastructure for targeting repositories, opening PRs, reviews, and production merges. A small internal CLI delegates prompts to agents, runs formatting and linting via local MCP, evaluates diffs with an LLM judge, uploads logs to GCP, captures MLflow traces, and can switch between agents/LLMs. The agent is also exposed through MCP so Slack/GitHub users can start ad hoc background coding tasks.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  fleet-management[Fleet Management] --> fleetshift-prompt[Fleetshift prompt]
  fleetshift-prompt --> internal-cli[Internal CLI]
  internal-cli --> coding-agent[Coding agent]
  internal-cli --> local-mcp-tools[Local MCP tools]
  internal-cli --> llm-judge[LLM judge]
  internal-cli --> gcp-logs[GCP logs]
  internal-cli --> mlflow-traces[MLflow traces]
  coding-agent --> pull-request-pipeline[Pull request pipeline]
  fleet-management --> pull-request-pipeline
  pull-request-pipeline --> human-reviewers[Human reviewers]
  slack-and-github-entrypoints[Slack and GitHub entrypoints] --> interactive-agent[Interactive agent]
  interactive-agent --> coding-agent
  slack-and-github-entrypoints --> human-reviewers

Components

Data flow

Engineers define a Fleetshift prompt or interact through Slack/GitHub. Fleet Management selects target repositories and invokes Spotify's internal CLI. The CLI delegates work to a pluggable coding agent, runs formatting/linting through MCP, judges diffs with an LLM, uploads logs, captures MLflow traces, and hands successful changes back to the existing PR/review/merge pipeline.

Why this architecture

Spotify kept the mature Fleet Management shell and replaced only the brittle transformation declaration with an agent prompt. This reduces migration authoring friction while preserving targeting, PR creation, review, traceability, and merge controls. The internal CLI keeps agent/model choice pluggable and integrates verification, logs, and traces without exposing platform details to users.

Models used: Honk · Claude Code agents · LLM judge · pluggable LLMs

Scale

• Spotify had merged more than 1,500 AI-generated pull requests.
  “This post explores how we have evolved our Fleet Management platform with background coding agents and what more than 1,500 merged AI-generated pull requests have taught us about the future of large-scale software maintenance.”article, stated
• Around half of Spotify PRs had been automated by Fleet Management since mid-2024.
  “Since mid-2024, around half of Spotify’s pull requests have been automated by this system. ![Background coding agent_graph 1.”article, stated
• A deterministic Maven updater exceeded 20,000 lines of code.
  “While its core function — identifying pom.xml files and updating Java dependencies — is straightforward, handling all corner cases has led to the transformation script growing to over 20,000 lines of code.”article, stated
• Spotify reports 60-90% time savings for these migrations.
  “For these migrations we’ve seen a total time saving of 60–90% compared to writing the code by hand the good old-fashioned way.”article, stated
• Hundreds of developers interact with the agent.
  “We see great impact and momentum from our first iteration of tools: hundreds of developers now interact with our agent, and we’ve already merged more than 1500 pull requests.”article, stated

Alternatives considered (roads not taken)

Continue writing deterministic migration scripts

Rejected because: Complex transformations required specialized expertise and could grow into very large scripts.

Adopt an off-the-shelf coding agent directly

Rejected because: Spotify wanted a preconfigured CLI with local MCP checks, LLM judge, logs, traces, and agent/model pluggability.

Replace the whole Fleet Management pipeline

Rejected because: Targeting, PR creation, review, and production merge infrastructure already worked.

Patterns

tool-use-orchestration (developer-platform)

The internal CLI coordinates coding agents, MCP checks, LLM judging, GCP logs, MLflow traces, and PR workflows.

llm-as-judge (verification)

An LLM judge evaluates generated diffs as part of the internal CLI workflow.

observability-tracing (operations)

Agent runs upload logs to GCP and capture traces in MLflow.

model-routing (model-ops)

The CLI can switch between different agents and LLMs as the ecosystem changes.

human-in-the-loop-gate (review)

Generated changes still flow through Spotify's PR review and merge process.

Tradeoffs

Replace only the transformation declaration

Chose agent prompts inside existing Fleet Management over rebuilding migration orchestration end to end.

Rationale: The existing platform already handled targeting, PRs, reviews, and production merges reliably.

Build a Spotify-specific CLI

Chose small internal CLI with checks, judge, logs, traces, and pluggability over off-the-shelf agent as-is.

Rationale: The CLI lets Spotify integrate agent work into existing developer workflows and swap technologies.

Expose agents via MCP for ad hoc work

Chose Slack/GitHub entrypoints with an interactive agent over only scheduled fleet migrations.

Rationale: The same agent infrastructure can support ADR capture, product-manager changes, and other background tasks.

Gotchas (what bit them)

Deterministic transformations hit a complexity wall.

Root cause: AST/regex migrations require specialized expertise and extensive corner-case code.

Resolution: Spotify uses prompts and coding agents for more complex code changes while retaining Fleet Management orchestration.

Background agents introduce performance and predictability risk.

Root cause: Agents can take a long time and produce unpredictable output.

Resolution: Spotify focuses on validation, quality control, guardrails, sandboxing, and feedback-loop posts.

Safety and cost become platform concerns at LLM scale.

Root cause: Large-scale LLM agent runs are computationally expensive and can make broad code changes.

Resolution: Spotify adds guardrails, sandboxing, quotas/traces, and validation mechanisms around agent runs.

Evaluation

Spotify evaluates the system with merged PR counts, time-savings estimates, adoption across developers, PR review outcomes, LLM-judge checks, logs/traces, and observed tradeoffs around performance, predictability, safety, and cost.

Metrics: more than 1,500 merged AI-generated PRs · around half of PRs automated by Fleet Management since mid-2024 · 60-90% time savings for migrations · over 20,000 lines in a deterministic Maven updater · hundreds of developers using the agent

Outcomes

• Spotify merged more than 1,500 AI-generated pull requests into production.
• The agent platform supports language modernization, breaking-upgrade migrations, UI component migrations, and config changes.
• Reported migration time savings are 60-90% versus hand-written code.
• The same infrastructure now supports ad hoc background coding tasks from Slack and GitHub Enterprise.

The case gives strong production scale: >1,500 merged AI PRs, hundreds of developers, about half of PRs automated by Fleet Management since mid-2024, and 60-90% time savings on migrations.