Large-Scale Learned Retrieval System with Two-Tower Architecture

pinterest · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Large-scale learned retrieval for recommendations

Pinterest needed to modernize candidate generation for homefeed and notifications by moving from heuristic retrieval to learned embedding-based retrieval over billions of pins. The system had to serve more than 500 million monthly active users, compute fresh user embeddings online, precompute item embeddings offline, support periodic retraining, avoid mismatched embedding spaces across separately deployed services, and improve engagement enough to consolidate existing candidate generators.

Constraints:

• retrieval starts from billions of Pin candidates and narrows to thousands before ranking
• Pinterest serves over 500 million monthly active users
• previous retrieval systems relied on heuristics such as Pin-Board graphs and followed interests
• softmax over the full item corpus is computationally infeasible
• user and item towers produce separate model artifacts deployed to separate services
• the item indexing pipeline usually takes longer than the viewer model to become ready
• partial ANN index rollouts and rollbacks must not cause embedding-space mismatches

Architecture

Pinterest built a two-tower learned retrieval system trained from logged user engagement events. The user tower encodes long-term engagement, profile, context, and real-time sequence features, while the item tower computes Pin embeddings. Training uses sampled softmax with in-batch negatives and logit correction for item popularity bias. Serving splits online user-embedding computation from offline item-embedding indexing: the homefeed backend computes the user embedding at request time, then queries Pinterest's Manas ANN serving system over precomputed item embeddings. Auto-retraining periodically refreshes the model and validates performance before deployment. Each ANN host carries model-version metadata generated with the index so the backend computes user embeddings using the matching model version, including during rolling index deployments and rollback.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  events["Logged engagement events"] --> model["Two-tower retrieval model"]
  model --> user["User tower"]
  model --> item["Item tower"]
  item --> index["Offline indexing pipeline"]
  index --> manas["Manas ANN serving system"]
  index --> metadata["Model version metadata"]
  metadata --> manas
  backend["Homefeed backend"] --> metadata
  backend --> user
  backend --> manas
  backend --> funnel["Recommendation funnel"]
  funnel --> generator["Learned retrieval candidate generator"]
  retrain["Auto-retraining workflow"] --> model
  retrain --> index
  retrain --> generator

Components

Data flow

Logged engagement events train a two-tower retrieval model. The item tower precomputes millions of item embeddings and pushes them into Manas for ANN search. At request time, the backend computes a user embedding from long-term engagement, profile, context, and real-time sequence signals, asks the assigned ANN host for its model-version metadata, loads the corresponding user tower, and retrieves candidates from the matching item-embedding index. Retrieved candidates flow into Pinterest's multi-stage recommendation funnel, where lightweight scoring and ranking narrow billions of candidate pins into personalized feeds.

Why this architecture

Pinterest uses two towers because independent user and item embeddings make nearest-neighbor retrieval cheap enough for online serving while still allowing personalized request-time user features. Offline item indexing avoids recomputing millions of item embeddings per request. The model-version metadata exists because two-tower systems create separate user-model and item-index artifacts; without synchronization, partial rollouts or rollback would make the user embedding and item embedding spaces incompatible.

Models used: two-tower retrieval model · user tower · item tower · PinnerSage · sequence transformer · transformer-based ranking model · Manas ANN serving system · HNSW

Scale

• Pinterest serves over 500 million monthly active users.
  “Pinterest serves over 500 million MAUs”article, stated
• The retrieval funnel starts from billions of candidate pins.
  “retrieving candidates from billions of candidate pools”article, stated
• The multi-stage funnel narrows billions of Pin candidates to thousands before ranking.
  “retrieve candidates from billions of Pin content corpus and narrow it down to thousands of candidates”article, stated
• The offline indexing pipeline computes millions of item embeddings.
  “millions of item embeddings are computed and pushed to our in-house Manas serving system”article, stated
• Homefeed has more than 20 production candidate generators.
  “It has over 20 candidate generators served in production with different retrieval strategies.”article, stated
• The learned retrieval candidate generator has top user coverage and top-three save rates.
  “It has the top user coverage and top three save rates.”article, stated

Alternatives considered (roads not taken)

Continue heuristic retrieval based on Pin-Board graphs or followed interests

Rejected because: Pinterest built an embedding-based system learned from logged user engagement events to improve candidate generation.

Compute softmax over the entire item corpus

Rejected because: The full corpus contains billions of items, so Pinterest used sampled softmax with in-batch negatives.

Deploy user-tower and item-index artifacts independently without synchronization

Rejected because: Mismatched embedding spaces drastically reduce candidate quality, especially during index rollout or rollback.

Refresh models manually or infrequently

Rejected because: Pinterest says real-world recommendation systems need frequent retraining to capture recent user preferences and content trends.

Patterns

retrieval-ranking (retrieval)

Pinterest uses a learned embedding-based candidate generator before lightweight scoring and ranking in a multi-stage recommendation funnel.

latency-optimization (serving)

Pinterest splits serving into request-time user embeddings and offline item indexing so online retrieval can use nearest-neighbor search efficiently.

eval-driven-development (model-operations)

The auto-retraining workflow validates model performance before deploying refreshed model and index artifacts.

fallback-degradation (reliability)

Pinterest keeps the latest N user-tower versions so rollback of the ANN service can still compute embeddings in the matching space.

canary-rollout (deployment)

The version metadata supports partial ANN-host rollouts where some hosts run version N while others run N+1 without embedding-space mismatch.

Tradeoffs

Retrieval representation

Chose Embedding-based learned retrieval over Heuristic graph or interest-based retrieval.

Rationale: Pinterest wanted organic-content retrieval learned from logged engagement events and deployable across homefeed and notifications.

Training objective approximation

Chose In-batch negative sampled softmax over Softmax over the full item corpus.

Rationale: The full corpus is too large, so sampled negatives provide a memory-efficient training approximation.

Serving split

Chose Online user embeddings plus offline item indexing over Compute both sides online or precompute both sides only offline.

Rationale: Request-time user embeddings incorporate fresh user features, while offline item indexing keeps item retrieval scalable.

Version synchronization mechanism

Chose ANN-host metadata mapped to model versions over Assume the latest user model and latest item index are always compatible.

Rationale: Per-host metadata handles partial rollouts, different deployment speeds, and rollback without mismatched embedding spaces.

Gotchas (what bit them)

Two-tower retrieval creates a hidden version-coupling problem.

Root cause: The user tower and item tower become separate artifacts deployed to separate services, and the item index often takes longer to build than the viewer model.

Resolution: Generate per-ANN-host model-version metadata with the index and have the backend compute user embeddings with the corresponding model version.

Embedding-space mismatch can cause candidate quality to collapse.

Root cause: A user embedding from one model version may be searched against an item embedding index from another version.

Resolution: Use anytime synchronization so the user model version always matches the assigned ANN host's index version.

Training examples can teach popularity instead of relevance.

Root cause: Sampled training items reflect natural popularity bias in the training set.

Resolution: Apply logit correction based on estimated item probability.

Fresh recommendations require frequent retraining but frequent retraining increases deployment risk.

Root cause: User preferences and content trends change, while retrained model artifacts and indexes roll out at different speeds.

Resolution: Retrain periodically, validate performance before deployment, and coordinate model/index versions through metadata.

Evaluation

Pinterest evaluates the learned retrieval system through model-performance validation before deployment, production homefeed candidate-generator metrics, user coverage, save rate, site engagement impact, and infrastructure behavior during rollouts and rollback.

Metrics: model performance validation before deployment · top user coverage · top three save rates · overall site engagement wins · candidate quality during rollout · ability to deprecate two other candidate generators · rollback compatibility through latest N viewer model versions

Outcomes

• Pinterest deployed the learned retrieval system for homefeed and notifications.
• The candidate generator achieved top user coverage and top-three save rates among production generators.
• The learned retrieval system helped deprecate two other candidate generators.
• Pinterest reports huge overall site engagement wins, though the article does not disclose exact quantitative lift.
• The metadata synchronization design supports index rollouts and rollback without user/item embedding-space mismatch.

The system operates in Pinterest's recommendation funnel for over 500 million MAUs, retrieves from billions of Pin candidates, narrows to thousands before ranking, computes millions of item embeddings for Manas ANN serving, and participates in a homefeed system with over 20 production candidate generators.