Background Coding Agents for Large-Scale Dataset Migrations

spotify · Media & Entertainment · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Large-scale dataset consumer migration

Spotify needed to migrate thousands of downstream data-pipeline consumers from deprecated user datasets to new versions within six months without spending roughly ten engineering weeks on manual edits.

Constraints:

• two deprecated datasets had about 1,800 direct downstream pipelines
• several thousand more pipelines were indirectly affected
• three frameworks were involved: BigQuery Runner, dbt, and Scio
• Scio lacked enough standardization for comprehensive prompts at the time
• BigQuery Runner and dbt repositories often lacked build-time unit tests

Architecture

Spotify used Backstage endpoint lineage and Codesearch to identify affected repositories, Fleetshift to orchestrate the migration, and Honk background coding agents to generate PRs from carefully engineered context files. The team narrowed scope to the more standardized BigQuery Runner and dbt frameworks, encoded field mappings in tables, instructed Honk to leave judgment-call fields unchanged with comments and migration-guide links, and monitored 240 automated migration PRs through Fleetshift/Backstage while relying on owning teams for manual testing where build-time tests were absent.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  deprecated-datasets[Deprecated datasets] --> backstage-lineage[Backstage lineage]
  backstage-lineage --> codesearch[Codesearch]
  codesearch --> fleetshift[Fleetshift]
  context-file[Context file] --> honk-agent[Honk agent]
  fleetshift --> honk-agent
  honk-agent --> dbt-repositories[dbt repositories]
  honk-agent --> bigquery-runner-repositories[BigQuery Runner repositories]
  context-file --> scio-repositories[Scio repositories]
  honk-agent --> automated-prs[Automated PRs]
  automated-prs --> downstream-owning-teams[Downstream owning teams]
  fleetshift --> migration-dashboard[Migration dashboard]
  migration-dashboard --> downstream-owning-teams

Components

Data flow

Backstage lineage surfaces downstream dataset consumers. Codesearch finds target repositories across Spotify GitHub Enterprise and Fleetshift marks them in scope. Honk receives comprehensive context files and migration mappings, edits standardized dbt/BigQuery Runner repositories, leaves ambiguous fields commented for humans, opens PRs, and Fleetshift/Backstage provides progress monitoring and links to PRs for downstream teams.

Why this architecture

Spotify combined lineage, repository search, orchestration, and background coding because the hard part was not just editing code: it was scoping thousands of consumers, providing enough context for a constrained agent, and managing PR progress across owning teams. The team deliberately limited scope to standardized frameworks and left use-case-specific judgment calls for humans.

Models used: Honk · Claude Code

Scale

• The deprecated datasets had about 1,800 direct downstream pipelines.
  “These deprecated datasets had ~1,800 direct downstream data pipelines between them and indirectly impacted several thousand more across the entire company.”article, stated
• The migration had a six-month deadline.
  “We faced the prospect of migrating ~1,800 direct downstream data pipelines in only six months, across three very different pipeline frameworks that we use at Spotify: the SQL-based BigQuery Runner and dbt frameworks, and the Scala-based Scio.”article, stated
• Manual work was estimated at around ten engineering weeks.
  “We estimated that it would have taken around 10 engineering weeks of effort to complete these migrations manually.”article, stated
• Spotify rolled out 240 automated migration PRs.
  “That said, we successfully rolled out 240 automated migration PRs using [Fleetshift](https://backstage.spotify.”article, stated

Alternatives considered (roads not taken)

Manually migrate all downstream pipelines

Rejected because: The estimate was about ten engineering weeks for a six-month, 1,800-pipeline effort.

Automate Scio migrations in the first pass

Rejected because: Scio's flexibility made comprehensive prompting too unwieldy without external skills/configurability.

Repurpose a human migration guide directly

Rejected because: The generated context omitted mapping detail and caused incorrect agent assumptions.

Patterns

tool-use-orchestration (developer-platform)

Backstage, Codesearch, Fleetshift, and Honk coordinate scoping, editing, PR creation, and monitoring.

human-in-the-loop-gate (review)

Ambiguous fields are left for human reviewers, and owning teams manually test PRs where automated tests are absent.

context-compaction (prompting)

The team compresses migration rules and mappings into context files the agent can use without external access.

eval-driven-development (process)

The team learned that validation and standardized data/framework conventions are prerequisites for scaling background agents.

Tradeoffs

Limit Honk's access during migration

Chose comprehensive up-front context files over allowing the agent to read schemas/docs via MCP or skills.

Rationale: The design established guardrails around possible migration outcomes.

Leave judgment-call fields unchanged

Chose comments with links to human guides over having Honk make use-case-specific migration decisions.

Rationale: Human owners were better positioned to decide ambiguous field migrations.

Use Backstage/Fleetshift monitoring

Chose central overview UI and PR links over manually searching repositories for migration progress.

Rationale: Ongoing troubleshooting and team communication are part of large migration success.

Gotchas (what bit them)

Flexible pipeline frameworks break all-in-one prompting.

Root cause: Scio varied too much across teams to capture all permutations in a comprehensive prompt.

Resolution: Spotify deferred Scio and focused on more standardized dbt and BigQuery Runner repositories.

Human-oriented migration guides are not automatically agent-ready.

Root cause: They may leave implicit mapping knowledge that an agent will guess incorrectly.

Resolution: Spotify rewrote mappings into explicit tables and fine-grained instructions.

Lack of build-time tests removes a key feedback loop.

Root cause: BigQuery Runner and dbt repositories rarely used build-time unit testing.

Resolution: Downstream owning teams had to perform manual testing before merge.

Evaluation

Spotify evaluated the migration through scope discovery, PR rollout count, manual-vs-automated effort, framework suitability, context-file iteration, manual test/review by owning teams, and Fleetshift monitoring.

Metrics: about 1,800 direct downstream pipelines · six-month migration window · around 10 engineering weeks avoided · 240 automated migration PRs · three pipeline frameworks considered, two automated initially

Outcomes

• Spotify rolled out 240 automated dataset migration PRs.
• The project saved an estimated ten engineering weeks.
• The team learned that standardized data frameworks and validation are critical for background coding agents.
• Future Honk work aims to let the agent gather context from JIRA or documentation before making code changes.

This is a production migration case with concrete scale: ~1,800 direct downstream pipelines, thousands more indirectly affected, six months, 240 PRs, and ten engineering weeks saved.