Large-Scale Personalization System Using LLMs for Buyer Profile Generation

etsy · E-commerce · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce personalization and buyer-profile generation

Etsy needed to personalize search and discovery for a marketplace with nearly 90 million buyers and more than 100 million listings. Traditional search and recommendation systems could not fully capture nuanced buyer tastes, shopping missions, seasonal behavior, and aesthetic preferences, so Etsy explored LLM-generated buyer profiles from browsing and purchasing behavior while keeping cost, privacy, refresh cadence, and downstream business measurement under control.

Constraints:

• the system serves nearly 90 million buyers and over 100 million listings
• input behavior includes searches, item views, purchases, and favorites from internal systems
• the initial profile-generation approach was too expensive and would take weeks to update at full scale
• the profile generator must avoid over-inferring interests when supporting data is weak
• personalization must respect privacy-by-design expectations and buyer opt-out requirements
• profile freshness must balance user interest drift against LLM and data-processing cost
• success must be measured through downstream search and shopping behavior, not just offline profile quality

Architecture

Etsy built an offline LLM personalization pipeline that retrieves buyer session and listing data from internal feature-store and BigQuery sources, prompts an LLM to generate structured buyer profiles, and uses those profiles for personalized query rewriting and refinement pills. The production architecture was made viable through data-source migration to clustered and partitioned BigQuery tables, token reduction from two years to nine months of sessions, larger batch sizes, managed concurrency, smaller-model prompt tuning, and Airflow DAG orchestration with staggered user-id batches.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  activity["Buyer activity sources"] --> prompt["Prompt assembly pipeline"]
  prompt --> llm["LLM profile generator"]
  cost["Cost optimization controls"] --> prompt
  cost --> llm
  airflow["Airflow profile DAG"] --> activity
  airflow --> llm
  llm --> store["Buyer profile store"]
  store --> rewrite["Query rewriting service"]
  store --> pills["Refinement pill service"]
  store --> refresh["Profile refresh monitor"]
  refresh --> airflow
  rewrite --> search["Search experience"]
  pills --> search
  search --> measure["Measurement framework"]
  measure --> refresh

Components

Data flow

The pipeline gathers recent buyer searches, item views, purchases, favorites, and listing information from Etsy internal data sources. That activity data is formatted into an LLM prompt, which produces a structured buyer profile containing inferred categorical interests, confidence values, explanations, and observed behaviors. Downstream search systems use the profile to rewrite queries with predicted style or category interests, and the web interface exposes high-confidence interests as refinement pills. Profile-refresh logic monitors engagement, drift, seasonality, and degradation signals so profiles can be updated without blindly regenerating every buyer at the same cadence.

Why this architecture

Etsy chose an offline profile-generation architecture because personalization needed rich behavior context and could be batch-refreshed rather than generated synchronously for every search. BigQuery-backed data access, token-window reduction, larger batches, and Airflow concurrency let the team convert an impractical weeks-long LLM workload into a scheduled production pipeline. Structured profiles also gave downstream systems a reusable personalization artifact for query rewriting, refinement pills, and future cold-start inheritance experiments.

Models used: large language models · smaller LLM for cost-optimized profile generation · collaborative filtering for inheritance profiles

Scale

• Etsy serves nearly 90 million buyers.
  “Every day, shoppers from Etsy's community of nearly 90M buyers visit our marketplace”article, stated
• The personalization problem spans more than 100 million listings.
  “But with over 100 million listings, how do we help each buyer find exactly what they're looking for?”article, stated
• The optimized pipeline reduced profile generation from 21 days to 3 days for 10 million users.
  “These improvements reduced buyer profile generation time dramatically — from 21 days down to 3 days for 10 million users.”article, stated
• Cost optimization reduced estimated cost by 94% per million users.
  “Through these various cost management techniques, we were able to reduce the estimated cost by 94% per million users.”article, stated
• Input history was reduced from about two years to nine months of session data.
  “Initially, we were including about 2 years of session data. Now, we’ve reduced this to just the last 9 months.”article, stated

Alternatives considered (roads not taken)

Use listing API endpoints as the main data source

Rejected because: Etsy moved to clustered and partitioned BigQuery tables to make large-scale profile generation efficient.

Feed about two years of session data into each prompt

Rejected because: The larger window inflated tokens and holiday-shopping corrections, so Etsy reduced context to nine months.

Use a larger or more expensive model for profile generation

Rejected because: Prompt adjustments let Etsy keep high quality while moving to a smaller model and reducing cost.

Regenerate every buyer profile with a fixed cadence

Rejected because: Etsy instead describes dynamic refresh timing based on activity, drift, seasonality, and degradation signals.

Leave new users without profiles until enough history accumulates

Rejected because: Etsy explored inherited profiles through collaborative filtering for users with minimal behavior.

Patterns

batch-offline-processing (workflow)

Etsy generates buyer profiles through scheduled offline batches rather than per-request LLM calls, using Airflow, larger batch sizes, and managed concurrency.

cost-aware-routing (operations)

Etsy redesigned data access, prompt context, model selection, batching, and compute allocation to reduce the cost of LLM profile generation at marketplace scale.

query-rewriting (retrieval)

Buyer profiles personalize backend search by enriching underspecified queries with predicted style and product-interest terms.

eval-driven-development (evaluation)

Etsy measures profile-based personalization through CTR, conversion, refinement-pill engagement, query-reformulation success, and degradation signals.

privacy-preserving-processing (governance)

The buyer-profile work is framed around anonymous profiles, privacy-by-design, and buyer privacy considerations for behavioral data.

Tradeoffs

Profile generation cadence

Chose offline Airflow-based batching and staggered user-id processing over synchronous or unrestricted generation across the full buyer base.

Rationale: Batch orchestration enables large-scale profile refreshes while managing BigQuery and OpenAI rate limits.

Behavior-history window

Chose nine months of session data over about two years of session history.

Rationale: The shorter window lowers token volume and prompt-correction burden while retaining enough recent behavior for personalization.

Profile confidence behavior

Chose empty interest/confidence fields when evidence is insufficient over forcing the LLM to produce categorical interests for every buyer.

Rationale: Etsy explicitly constrains the structured profile so weak behavioral evidence does not become overconfident personalization data.

Cold-start strategy

Chose experimenting with inheritance profiles via collaborative filtering over waiting for every new user to accumulate extensive session history.

Rationale: Matching sparse early behavior to similar existing profiles could provide personalization before enough direct history exists.

Gotchas (what bit them)

Naive LLM personalization was operationally infeasible at Etsy scale.

Root cause: Generating profiles for roughly 90 million buyers would have taken weeks and cost too much before data, token, model, and batching optimizations.

Resolution: Move listing data to efficient BigQuery tables, reduce prompt tokens, increase batch sizes, manage concurrency, and use smaller models where quality holds.

Too much session history made prompts heavier and introduced seasonal bias.

Root cause: Two years of behavior inflated token volume and required more correction for holiday shopping effects.

Resolution: Reduce the input window to nine months, which lowered tokens and reduced holiday-shopping prompt corrections.

Buyer profiles should not invent categorical interests from weak behavior.

Root cause: Some users do not have enough supporting interaction data for confident categorical inferences.

Resolution: Allow categorical-interest and confidence fields to be empty while still recording explanations and observed interests.

Freshness and cost are in tension for profile maintenance.

Root cause: Profiles need to track changing interests, seasonal effects, and performance degradation, but regenerating profiles has nontrivial cost.

Resolution: Use dynamic refresh timing based on user activity, drift detection, seasonal considerations, and engagement degradation signals.

Evaluation

Etsy frames evaluation around systematic experimentation and production business metrics: profile accuracy, CTR lift, conversion impact, refinement-pill engagement, search-query reformulation success, and degradation thresholds for profile refresh.

Metrics: profile accuracy against actual user interests and shopping behaviors · click-through-rate lift · conversion-rate impact · refinement-pill engagement · search-query reformulation success · personalization degradation below engagement thresholds

Outcomes

• profile generation time dropped from 21 days to 3 days for 10 million users
• estimated cost fell by 94% per million users
• buyer profiles enabled personalized query rewriting and refinement-pill experiments
• the work remained exploratory rather than described as a fully rolled-out personalization replacement

The system targets Etsy-scale personalization across nearly 90 million buyers and more than 100 million listings, with a measured 10-million-user processing benchmark.