Generating 1.4 Billion Personalized Music Narratives for Wrapped Archive

spotify · Media & Entertainment · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: personalized content generation at global launch scale

Spotify needed to generate personalized Wrapped Archive narratives for hundreds of millions of users, grounded in real listening data, while keeping the stories creative, safe, non-repetitive, cost-feasible, concurrency-safe, and ready for a single global launch moment.

Constraints:

• each eligible user could receive up to five personalized reports
• the system had to pre-generate roughly 1.4 billion reports before launch day
• each report required a model call under strict latency and capacity constraints
• creative narratives had to avoid hallucination, stereotyping, unsafe topics, and off-brand tone
• multiple reports for the same user could be generated and written concurrently
• Wrapped traffic arrives as a global spike rather than a gradual ramp
• human review could not cover more than a billion reports

Architecture

Spotify built an offline generation system for Wrapped Archive. A distributed data pipeline ranked each user's listening days by narrative potential, stored up to five remarkable days and relevant listening history in object storage, then published work to a pubsub queue. A fine-tuned production model generated one report at a time per user, using previous reports to avoid repetition, while reports were persisted in a column-oriented key-value store designed for concurrent day-level writes. Separate evaluation, logging, remediation, capacity-planning, and launch-warming systems made the billion-report batch safe to generate and serve.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  listening_history_events["Listening history events"] --> remarkable_day_heuristic_pipeline["Remarkable-day heuristic pipeline"]
  remarkable_day_heuristic_pipeline --> object_storage_snapshots["Object storage snapshots"]
  object_storage_snapshots --> pubsub_generation_queue["Pubsub generation queue"]
  prompt_and_prototype_tooling["Prompt and prototype tooling"] --> fine_tuned_production_model["Fine-tuned production model"]
  prompt_and_prototype_tooling --> human_review_loop["Human review loop"]
  pubsub_generation_queue --> fine_tuned_production_model
  fine_tuned_production_model --> report_content_table["Report content table"]
  fine_tuned_production_model --> evaluation_data_warehouse["Evaluation data warehouse"]
  report_content_table --> completion_metadata_columns["Completion metadata columns"]
  report_content_table --> serving_path["Serving path"]
  completion_metadata_columns --> serving_path
  evaluation_data_warehouse --> llm_as_judge_evaluators["LLM-as-judge evaluators"]
  evaluation_data_warehouse --> remediation_loop["Remediation loop"]
  llm_as_judge_evaluators --> remediation_loop
  human_review_loop --> remediation_loop
  remediation_loop --> fine_tuned_production_model
  remediation_loop --> report_content_table
  serving_path --> wrapped_users["Wrapped users"]

Components

Data flow

Listening history is first reduced into candidate remarkable days by priority-ordered heuristics. The selected day data and listening context are stored in object storage and published asynchronously to a queue. A fine-tuned model generates report text, with earlier reports available as context for later reports for the same user. Report content is written before lightweight completion metadata in a column-oriented database. Generated reports are copied into an evaluation warehouse, judged by larger models and targeted human review, and problematic cohorts can be deleted, fixed upstream, and replayed before the pre-scaled serving path handles the Wrapped launch spike.

Why this architecture

Spotify separated candidate-day computation, offline generation, storage, evaluation, remediation, and launch serving because every part had a different scaling risk. Pre-generation avoided launch-time model latency, distillation made a billion model calls economically feasible, column-oriented writes removed coordination from concurrent report completion, and automated evaluation/replay handled quality failures that human review could not inspect at full corpus scale.

Models used: LLM · frontier model · fine-tuned production model · larger models acting as judges

Scale

• Spotify identified up to five remarkable days for each eligible user.
  “For each eligible user, we identified up to five remarkable days from their 2025 listening history.”article, stated
• The generation target was roughly 350 million eligible users and 1.4 billion reports.
  “Around 350 million eligible users, each getting up to five reports, totaling roughly 1.4 billion reports, all pre-generated before Wrapped launch day.”article, stated
• Generation had to sustain thousands of model requests per second for days.
  “That meant sustaining thousands of requests per second for days, under strict latency constraints.”article, stated
• The initial generation pass ran continuously for four days.
  “For four days straight, the generation engine ran without pause!”article, stated
• At this scale, even a 0.1% failure rate would create millions of broken stories.
  “even a 0.1% failure rate would translate to millions of “broken” stories.”article, stated
• Spotify evaluated a random sample of about 165,000 reports rather than the full corpus.
  “we evaluated a large random sample (~165,000 reports) rather than the full corpus.”article, stated
• The launch was a global big-bang event where millions of users could arrive at once.
  “One second the service is idle; the next, millions of users are hitting it.”article, stated

Alternatives considered (roads not taken)

Use high-performance frontier models for every production report

Rejected because: The article says frontier models were useful for prototyping but economically infeasible for more than a billion report generations.

Track completed days as a serialized list in a user row

Rejected because: A read-modify-write list would have risked lost updates when multiple day reports completed concurrently for the same user.

Depend on reactive autoscaling during Wrapped launch

Rejected because: Wrapped traffic spikes instantly, so Spotify pre-scaled and warmed infrastructure before real traffic arrived.

Evaluate every generated report with model judges

Rejected because: To preserve efficiency, Spotify evaluated a large random sample instead of the full 1.4 billion-report corpus.

Use one massive evaluation prompt

Rejected because: Spotify split judging into smaller rule-based queries to reduce non-determinism and allow parallel grading.

Patterns

batch-offline-processing (generation architecture)

Spotify pre-generated all Wrapped Archive reports offline before launch, using object storage, a pubsub queue, and asynchronous processing to run the billion-report workload ahead of user traffic.

model-distillation (model optimization)

Spotify used frontier-model outputs to create a reviewed gold dataset, then fine-tuned a smaller production model and applied DPO to make generation economically feasible.

eval-driven-development (quality and safety loop)

Prompting, LLM-as-judge evaluations, human review, structured logging, and remediation all fed back into prompt and generation changes.

llm-as-judge (automated evaluation)

Spotify used larger models as judges to grade generated reports across accuracy, safety, tone, and formatting dimensions.

human-in-the-loop-gate (creative and safety review)

Human review shaped prompt iteration, curated the distillation/preference data, and participated in targeted remediation for problematic reports.

guardrails-io-filtering (prompt safety controls)

The system prompt encoded a creative contract that required data-grounded storytelling and excluded unsafe or off-brand content categories.

observability-tracing (generation monitoring and fault isolation)

Spotify used real-time dashboards, structured logging, report IDs, and full metadata to monitor generation and trace corpus-level failures back to upstream causes.

durable-execution (replay and recovery)

After detecting bad outputs, Spotify could delete affected reports, fix the upstream issue or guardrail, and replay generation safely for those reports.

prompt-versioning (prompt iteration)

Spotify treated prompt design as a daily multi-month practice, comparing prompt versions and edge cases through prototype tooling.

Tradeoffs

Use a distilled production model for bulk generation.

Chose A smaller fine-tuned model trained from reviewed frontier-model references over Running high-performance frontier models for every report.

Rationale: The frontier model was good for prototyping, but the economics of over a billion production calls required a smaller faster model.

Pre-generate reports before launch.

Chose An offline batch generation engine over Generating each narrative on-demand during Wrapped traffic.

Rationale: Launch arrives as a global spike and each report requires a model call, so pre-generation moved model latency and throughput risk out of the user-facing path.

Generate one report at a time per user.

Chose Sequential per-user report generation with previous reports as context over Fully independent generation of all reports for a user.

Rationale: Earlier reports informed later reports so the generated stories could avoid repetition across a user's archive.

Model completion metadata as independent per-day cells.

Chose Column qualifiers by date within a user's row over A serialized completed-days list updated through read-modify-write.

Rationale: Independent cells let concurrent report completions land without locks or coordination, avoiding lost updates.

Sample for model-based evaluation.

Chose A large random sample of about 165,000 reports over LLM-judge evaluation of the entire corpus.

Rationale: Sampling preserved the efficiency gains from distillation while still giving Spotify corpus-level QA signal.

Pre-scale and warm infrastructure before launch.

Chose Pre-scaled compute/database capacity plus synthetic regional load tests over Waiting for autoscaling to react to observed demand.

Rationale: Wrapped traffic spikes too abruptly for reactive scaling, so Spotify warmed compute, caches, database tablets, and provider capacity before real users arrived.

Gotchas (what bit them)

A tiny failure rate becomes millions of bad experiences at Wrapped scale.

Root cause: The corpus had more than a billion reports, so even 0.1% bad output would affect millions of users.

Resolution: Spotify built automated evaluation, sampled model judging, structured logging, targeted human review, and remediation loops instead of relying on manual review.

Creative LLM output could hallucinate, stereotype, or drift off-brand.

Root cause: The reports were personalized narratives, so factual grounding, emotional tone, and safety constraints all had to hold simultaneously.

Resolution: Spotify split the prompt into a system-level creative contract and a user prompt that supplied detailed listening logs, stats, prior reports, and country context.

Model math weakness could corrupt personalized narratives.

Root cause: The generation input included raw listening logs, but the model was not trusted to compute listening statistics correctly.

Resolution: Spotify included a summarized stats block in the prompt so the model could narrate rather than calculate.

Concurrent report writes for the same user risked lost completion state.

Root cause: Each user could have up to five reports generated independently, so multiple writes for the same row could arrive nearly simultaneously.

Resolution: Spotify modeled each remarkable day as its own column qualifier, avoiding locks and read-modify-write updates.

Reactive autoscaling would be too slow for a global Wrapped launch.

Root cause: Traffic jumps from idle to millions of users at once, leaving no gradual ramp for autoscaling signals.

Resolution: Spotify pre-scaled compute and database capacity, coordinated model-provider capacity, and ran synthetic regional load tests to warm compute and storage.

The model faithfully amplified an upstream data bug into confident wrong stories.

Root cause: A subtle timezone bug surfaced the wrong top discovery day, and the model generated a compelling narrative from that incorrect input.

Resolution: Structured evaluations and report metadata let Spotify trace the bug, quantify affected reports, fix the pipeline, bulk-delete, and replay safely.

More prompt instructions could reduce creative quality.

Root cause: Stacking constraints on the generator made output less creative even when instructions were well-intentioned.

Resolution: Spotify treated prompt and evaluation design as a coupled loop and learned to keep prompts lean enough to preserve story quality.

Evaluation

Spotify combined prompt-version comparisons, LLM-as-judge grading, human review, structured logging, corpus queries, regex-based matching, and replay. Larger judge models graded sampled production reports for accuracy, safety, tone, and formatting, while targeted review and metadata tracing drove remediation.

Metrics: accuracy · safety · tone · formatting · preference parity with the original baseline · throughput · system reliability · projected completion time · missing reports · data inconsistencies

Outcomes

• Spotify generated roughly 1.4 billion personalized Wrapped Archive reports before launch.
• The generation engine ran continuously for four days and completed an initial pass before remediation.
• A fine-tuned production model reached strong preference parity with the original baseline.
• Column-oriented storage allowed concurrent day-level writes without locks or read-modify-write cycles.
• Structured evaluation found a timezone bug, enabling Spotify to trace, quantify, fix, delete, and safely replay affected reports.
• Wrapped Archive reached hundreds of millions of users globally.

The project operated at unusually high batch and launch scale: roughly 350 million eligible users, up to five reports per user, around 1.4 billion generated reports, thousands of model requests per second for days, and a single global launch moment where elevated latency could affect millions.