Personalized Music Recommendation at Scale Using LLMs and User Embeddings

spotify · Media & Entertainment · 2026 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Personalized recommendation systems with LLMs and user embeddings

Spotify needed to move from siloed recommendation models toward a unified, steerable, LLM-compatible recommendation architecture that can personalize across hundreds of millions of users and a massive catalog of tracks, podcasts, audiobooks, and videos.

Constraints:

• Traditional recommender systems used separate product-specific models for home ranking, playlists, search, podcasts, and ads.
• The system must operate across hundreds of millions of users and a catalog with more than 100 million tracks.
• Open-weight LLMs have world knowledge but do not natively understand Spotify's catalog IDs or user histories.
• The LLM cannot be trained individually on every Spotify user.
• Continual training on Spotify data risks catastrophic forgetting.
• Daily user-embedding generation is described as massive and expensive.

Architecture

Spotify combines transformer-based user representations, semantic IDs for catalog items, continual LLM training, and soft-token user personalization. User interaction history and request context become embeddings; content vectors for tracks, artists, podcasts, and episodes are compressed into hierarchical semantic ID tokens; open-weight LLMs are fine-tuned to speak those semantic IDs; and user embeddings are projected into the LLM token space as soft tokens so production recommendation features such as AI DJ, prompted playlists, taste profiles, and next-episode podcast recommendations can be both personalized and steerable.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  User_interaction_history["User interaction history"] --> Transformer_user_model["Transformer user model"]
  Transformer_user_model --> User_embedding_store["User embedding store"]
  Transformer_user_model --> Cross_content_embedding_space["Cross-content embedding space"]
  Catalog_embedding_pipeline["Catalog embedding pipeline"] --> Semantic_ID_tokenizer["Semantic ID tokenizer"]
  Catalog_embedding_pipeline --> Cross_content_embedding_space
  Semantic_ID_tokenizer --> Continual_LLM_training["Continual LLM training"]
  Open_weight_LLM_backbone["Open-weight LLM backbone"] --> Continual_LLM_training
  Continual_LLM_training --> Generative_recommender["Generative recommender"]
  User_embedding_store --> Soft_token_projection["Soft token projection"]
  Soft_token_projection --> Generative_recommender
  Generative_recommender --> Product_surfaces["Product surfaces"]
  Product_surfaces --> Taste_profile_feedback_loop["Taste profile feedback loop"]
  Taste_profile_feedback_loop --> Generative_recommender
  Taste_profile_feedback_loop --> User_interaction_history

Components

Data flow

User interactions, request context, and product surface context are processed by transformer user-modeling pipelines into embeddings; catalog entities are embedded and compressed into semantic IDs; Spotify trains or adapts open-weight LLMs on semantic-ID sequences and Spotify interaction data; user embeddings are projected into soft tokens inserted into the prompt; the generative recommender predicts tracks, episodes, or playlists; taste-profile edits and user feedback feed back into the generative model.

Why this architecture

Spotify's architecture bridges platform-specific recommendation knowledge with LLM world knowledge. User embeddings summarize taste at scale, semantic IDs teach LLMs the catalog in token form, and soft tokens personalize generation without training a separate model for every user.

Models used: Transformer user models · Autoencoder user models · Open-weight LLMs · Llama · Qwen · Spotify semantic ID models · Soft token projection model

Scale

• Spotify reports about 750 million users.
  “we have like about 750 million users right now”transcript, stated
• Spotify reports a catalog of more than 100 million tracks.
  “we have like a catalog of like 100 million plus tracks”transcript, stated
• Spotify reports around 400,000 audiobooks.
  “we have I think about 350 I think it's more like 400,000 audiobooks now”transcript, stated
• Spotify operates in about 184 markets.
  “we're in about 184 markets”transcript, stated
• Spotify has used machine learning models for at least a decade, with Discover Weekly around 2015.
  “we've been using machine learning models for at least like a decade if not more”transcript, stated

  “Discover Weekly or you're probably a user of that. Um that's been around since I think 2015”transcript, stated
• Spotify generates embeddings for more than a billion users every day.
  “we generate embeddings for like a billion plus users”transcript, stated

  “we do that every day, so it's it's like a massive uh pipeline. It's very expensive.”transcript, stated
• Semantic IDs compress a vector into roughly four to six tokens.
  “compresses that massive like let's say a thousand dimensional vector into like four or six tokens”transcript, stated
• The taste profile feature initially launched in only a few markets with expansion planned later in the year.
  “this is only supported like in a few markets but this is going to be expanded uh later this year”transcript, stated

Alternatives considered (roads not taken)

Keep siloed product-specific recommendation models

Rejected because: Spotify is moving from separate team-owned models toward a unified LLM-backbone model that supports user steerability.

Use traditional candidate generation and ranking only

Rejected because: The LLM-based generative approach adds steerability, natural-language interaction, explainability, and cross-content modeling.

Represent catalog content only with Spotify vectors

Rejected because: Spotify wants to combine internal catalog/user vectors with world knowledge from open-weight LLMs.

Train the LLM on every individual user

Rejected because: The model cannot be trained on all 750 million-plus users, so Spotify projects user embeddings into the LLM token space instead.

Ignore catastrophic forgetting during domain adaptation

Rejected because: Spotify acknowledges forgetting as a tradeoff when embedding platform knowledge into open-weight LLMs.

Patterns

context-engineering (context)

Spotify treats user interactions, request context, product surface, item context, semantic IDs, and soft tokens as modeled context for the LLM recommender.

agent-memory (personalization)

User embeddings and taste profiles encode long-term listening history and preferences so recommendations can be personalized at generation time.

model-routing (model-operations)

Spotify moves from product-specific recommender models toward a unified LLM backbone adapted with domain data, while still combining user, catalog, and LLM model components.

human-in-the-loop-gate (feedback)

The taste profile lets users inspect and edit what Spotify knows, then feeds those edits back into the generative recommendation model.

eval-driven-development (evaluation)

Spotify reports internal metrics and production rollout status for the LLM-based podcast next-episode recommender before expanding the approach.

Tradeoffs

Move from traditional candidate/ranking pipelines toward a generative recommender.

Chose Unified LLM-style sequential/generative model over Siloed candidate generation and ranking models only.

Rationale: The generative approach adds steerability and natural-language interaction while unifying recommendations across surfaces.

Use semantic IDs to teach catalog structure to the LLM.

Chose Hierarchical semantic ID tokens over Raw high-dimensional content vectors inside the LLM.

Rationale: Semantic IDs compress catalog vectors into token sequences the LLM can be trained to predict autoregressively.

Project user embeddings as soft tokens.

Chose Soft token personalization over Per-user LLM training.

Rationale: Soft tokens let the model personalize recommendations for a user without training on every individual user.

Accept continual-training tradeoffs for Spotify-specific recommendations.

Chose Fine-tune/open-weight LLM adaptation with Spotify data over Use open-weight LLM world knowledge unchanged.

Rationale: The adapted model can combine world knowledge with Spotify catalog knowledge, but the team must manage catastrophic forgetting.

Expose taste profiles to users.

Chose User-visible preference editing over Opaque personalization only.

Rationale: Taste profiles give users steerability and control over what Spotify remembers or forgets.

Gotchas (what bit them)

Siloed recommendation models fragment user understanding.

Root cause: Different Spotify product teams historically maintained separate models with different features and quality levels.

Resolution: Move toward a unified LLM-backbone recommendation model with shared user and catalog representations.

LLMs do not naturally understand Spotify catalog identifiers.

Root cause: Spotify's content knowledge exists as internal vectors and URIs, while open-weight LLMs understand language/world knowledge.

Resolution: Tokenize content vectors into semantic IDs and train the LLM to use those tokens.

Domain adaptation can cause catastrophic forgetting.

Root cause: Fine-tuning open-weight LLMs on Spotify catalog data may degrade some original world-knowledge capabilities.

Resolution: Monitor the tradeoff while using semantic IDs to combine platform knowledge with world knowledge.

User-scale personalization cannot be solved by training on each user individually.

Root cause: Spotify has hundreds of millions of users, and the LLM is trained on limited training data.

Resolution: Use collaborative filtering plus soft-token projection of user embeddings into the prompt.

The user-embedding pipeline is operationally expensive.

Root cause: Spotify generates embeddings for a billion-plus users every day.

Resolution: Use the embeddings as foundational infrastructure for downstream recommendation models and soft-token personalization.

Evaluation

The presentation does not disclose detailed metrics, but says the LLM-based recommender showed positive internal metrics and is productionized for next-episode podcast recommendations. The system is evaluated through internal recommendation metrics, production rollout, steerability/user-feedback features, and ongoing expansion across recommendation surfaces.

Metrics: Internal recommendation metrics · Productionization status · Next-episode recommendation quality · Steerability · Personalization quality · Taste profile edits · Podcast recommendation behavior · Cross-content embedding quality

Outcomes

• Spotify has productionized the LLM-based approach for next-episode podcast recommendations.
• AI DJ, prompted playlists, and taste profiles demonstrate user-facing steerability and natural-language recommendation surfaces.
• Semantic IDs let Spotify train LLMs to generate tracks, artists, or episodes as token sequences.
• Soft token projection lets user embeddings personalize LLM output without training on each user.
• The approach moves Spotify toward unified, steerable, generative recommendation across products.

The system is designed around about 750 million users, a 100M-plus-track catalog, about 400,000 audiobooks, millions of podcasts/video episodes, 184 markets, daily embeddings for a billion-plus users, semantic IDs of four to six tokens, and productionized podcast next-episode recommendations.