User Journey Identification Using LLMs for Personalized Recommendations

pinterest · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Personalized recommendation and user intent modeling

Pinterest needed to identify long-running user journeys from sparse activity data so recommendations and notifications could support user goals rather than only immediate interests.

Constraints:

Architecture

Pinterest built a hybrid journey-identification pipeline. A streaming inference system extracts keywords from searches, activity, and boards; embeds them with pretrained SearchSage text embeddings; clusters them hierarchically into journey candidates; ranks and diversifies journeys with traditional models; predicts journey stage; and increasingly uses LLMs for naming, expansion, relevance evaluation, and future consolidated inference. Offline LLM generation plus key-value storage keeps online serving efficient.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  user-activity-sources["User activity sources"] --> streaming-inference-pipeline["Streaming inference pipeline"]
  streaming-inference-pipeline --> keyword-extraction["Keyword extraction"]
  keyword-extraction --> searchsage-embeddings["SearchSage embeddings"]
  searchsage-embeddings --> hierarchical-clustering["Hierarchical clustering"]
  hierarchical-clustering --> journey-ranking-model["Journey ranking model"]
  searchsage-embeddings --> post-ranking-diversification["Post-ranking diversification"]
  journey-ranking-model --> post-ranking-diversification
  journey-ranking-model --> journey-stage-model["Journey stage model"]
  journey-ranking-model --> llm-journey-services["LLM journey services"]
  llm-journey-services --> safety-checks["Safety checks"]
  llm-journey-services --> qwen-fine-tuned-model["Qwen fine-tuned model"]
  qwen-fine-tuned-model --> ray-batch-inference["Ray batch inference"]
  streaming-inference-pipeline --> journey-serving-store["Journey serving store"]
  journey-ranking-model --> journey-serving-store
  journey-stage-model --> journey-serving-store
  safety-checks --> journey-serving-store
  ray-batch-inference --> journey-serving-store
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
User activity sources Provide searches, Pin actions, clickthroughs, repins, and boards. Streaming inference pipeline
Streaming inference pipeline Runs full and incremental journey inference for users. User activity sources, Keyword extraction, Journey serving store
Keyword extraction Dynamically extracts activity keywords and metadata. Streaming inference pipeline, SearchSage embeddings
SearchSage embeddings Embed keywords for clustering, ranking, and diversification. Keyword extraction, Hierarchical clustering, Post-ranking diversification
Hierarchical clustering Forms journey candidates from embedded keywords. SearchSage embeddings, Journey ranking model
Journey ranking model Scores candidate journeys using user, engagement, and recency features. Hierarchical clustering, Post-ranking diversification, Journey serving store
Post-ranking diversification Penalizes journeys too similar to higher-ranked journeys. SearchSage embeddings, Journey ranking model
Journey stage model Classifies journeys as situational or evergreen and predicts whether situational journeys ended. Journey ranking model, Journey serving store
LLM journey services Generate names, related journeys, relevance scores, labels, and future consolidated inference. Journey ranking model, Safety checks, Journey serving store, Qwen fine-tuned model
Qwen fine-tuned model Runs in-house LLM inference after GPT-generated labels bootstrap fine-tuning. LLM journey services, Ray batch inference
Ray batch inference Improves efficiency and scalability for full and incremental LLM inference. Qwen fine-tuned model, Journey serving store
Safety checks Filter generated journeys against Pinterest safety standards. LLM journey services, Journey serving store
Journey serving store Stores generated journeys and related journeys for efficient online serving. Streaming inference pipeline, Journey ranking model, LLM journey services, Safety checks, Ray batch inference

Data flow

User searches, Pin activity, and boards feed a streaming inference pipeline that extracts keywords and metadata. Pretrained embeddings represent those keywords, hierarchical clustering forms journey candidates, ranking and stage models score them, post-ranking diversification reduces monotony, LLMs name/evaluate/expand journeys, safety checks filter generated outputs, and journey-aware notifications are served through email and push channels.

3 supporting quotes
“The pipeline extracts keywords from multiple data sources including user search history, activity history (Pin closeups, repins, clickthroughs), and user boards.”summary, stated
“These keywords, along with associated metadata, are embedded using pretrained text embeddings and then subjected to hierarchical clustering to form journey clusters”summary, stated
“All generated journeys pass through safety checks to ensure they meet Pinterest's safety standards”summary, stated

Why this architecture

Pinterest used a hybrid pipeline because it lacked enough labeled data for a fully supervised journey system and needed to serve at massive scale. Traditional clustering/ranking kept online inference efficient, while LLMs were used where they added semantic lift: naming, expansion, evaluation, label generation, and the path toward consolidated future inference.

“Rather than attempting to build everything from scratch, the team adopted a pragmatic, incremental strategy that balanced the power of modern LLMs with the constraints of a production environment serving hundreds of millions of users.”summary, stated
“This demonstrates a sophisticated understanding of when to pay LLM inference costs (offline, for bounded generation tasks) versus when to use lighter-weight models (online, for ranking tasks).”summary, stated

Models used: SearchSage · GPT · Qwen · Point-wise ranking model · LLM relevance evaluator

Scale

Alternatives considered (roads not taken)

Use a predefined journey taxonomy

Rejected because: A fixed taxonomy would require maintenance, overlap with existing systems, and adapt slowly to trends.

“While a fixed taxonomy would offer consistency, they recognized it would risk overlapping with existing systems, require significant maintenance, and be slow to adapt to emerging trends.”summary, stated

Run all LLM inference online

Rejected because: Pinterest generated bounded related-journey data offline and used lighter ranking online to control latency and cost.

“they could use LLMs to generate this data offline and store it in a key-value store for efficient online serving.”summary, stated
“This reduces latency and cost but sacrifices some degree of personalization and adaptability.”summary, stated

Use only commercial LLMs in production

Rejected because: The team used GPT to generate labels, then fine-tuned Qwen for in-house scalable inference.

“use powerful commercial LLMs (GPT) for data generation and labeling, then fine-tune smaller, more cost-effective models (Qwen) that can be deployed at scale.”summary, stated

Replace all traditional components with a black-box LLM immediately

Rejected because: Pinterest kept ranking and stage models where simpler components were adequate and more interpretable.

“the team maintained traditional ML components for ranking and stage prediction where simpler models provide adequate performance and clearer interpretability.”summary, stated

Patterns

hybrid-search (retrieval)

Pinterest combined keyword extraction, pretrained embeddings, clustering, ranking, and diversification rather than relying on one retrieval signal.

“The pipeline extracts keywords from multiple data sources including user search history, activity history (Pin closeups, repins, clickthroughs), and user boards.”summary, stated
“These keywords, along with associated metadata, are embedded using pretrained text embeddings and then subjected to hierarchical clustering to form journey clusters”summary, stated

llm-as-judge (evaluation)

Pinterest uses LLMs to score predicted journeys for relevance, then validates those judgments against human assessments.

“they leverage LLMs to assess the relevance of predicted user journeys.”summary, stated
“Critically, they validated that LLM judgments closely correlate with human assessments in their use case before relying on this approach”summary, stated

synthetic-data-generation (data)

Pinterest used GPT to generate ground-truth labels for fine-tuning Qwen.

“They first tuned prompts and used GPT to generate ground truth labels for fine-tuning Qwen (an open-source LLM)”summary, stated

fine-tuning-adaptation (model operations)

A smaller open-source Qwen model was fine-tuned for cost-controlled in-house inference.

“fine-tune smaller, more cost-effective models (Qwen) that can be deployed at scale.”summary, stated

batch-offline-processing (infrastructure)

Bounded related-journey generation and Ray batch inference move expensive LLM work offline or into distributed batches.

“they could use LLMs to generate this data offline and store it in a key-value store for efficient online serving.”summary, stated
“The use of Ray for batch inference is a notable technical choice that reflects the growing maturity of the LLMOps tooling ecosystem.”summary, stated

safety-policy-enforcement (safety)

Generated journeys are filtered through safety checks before serving.

“All generated journeys pass through safety checks to ensure they meet Pinterest's safety standards - a critical production consideration often overlooked in research contexts.”summary, stated

Tradeoffs

Dynamic keyword extraction

Chose Extract user-activity keywords dynamically over Maintain a predefined journey taxonomy.

Rationale: Dynamic extraction adapts to personal behavior and new trends with less maintenance.

“The dynamic approach offered greater flexibility, personalization, and adaptability while leveraging existing infrastructure.”summary, stated

Offline LLM generation for related journeys

Chose Generate bounded related journeys offline and rank online over Perform all journey expansion with online LLM inference.

Rationale: This preserved serving efficiency while still using LLM semantic generation.

“This demonstrates a sophisticated understanding of when to pay LLM inference costs (offline, for bounded generation tasks) versus when to use lighter-weight models (online, for ranking tasks).”summary, stated

GPT-to-Qwen production path

Chose Use GPT for labels and fine-tuned Qwen for in-house inference over Serve commercial LLM calls directly at scale.

Rationale: The approach balanced label quality with cost control and deployment ownership.

“This two-stage approach is increasingly common in production LLMOps but requires significant ML infrastructure and expertise to execute effectively.”summary, stated

Hybrid ML plus LLM evolution

Chose Keep traditional models where they are efficient and interpretable while adding LLMs selectively over Move immediately to a pure LLM system.

Rationale: The hybrid system supported operational stability and debuggability while data and LLM capabilities matured.

“This hybrid approach allows them to understand and debug system behavior more effectively than a pure black-box LLM approach would.”summary, stated

Gotchas (what bit them)

New product categories may lack enough training data for fully supervised systems.

Root cause: Pinterest was identifying journeys before it had large labeled journey datasets.

Resolution: The team started lean with pretrained embeddings, a few hundred annotations, and incremental LLM adoption.

“The technical challenge was particularly interesting because Pinterest was building a new product category without substantial training data”summary, stated
“starting small with only a few hundred human-annotated examples”summary, stated

LLM inference costs can spiral in personalization systems.

Root cause: Journey inference has to run over very large user populations and remain fresh.

Resolution: Pinterest used offline generation, key-value stores, incremental inference, and lighter online ranking.

“This represents a thoughtful trade-off between freshness and computational efficiency - a common challenge in production LLMOps scenarios where inference costs can quickly spiral if not carefully managed.”summary, stated
“implementing both full inference for all users and incremental inference for recently active users to reduce overall inference costs.”summary, stated

LLM-based evaluation can import evaluator-model bias.

Root cause: An evaluator LLM may not match human judgment unless validated.

Resolution: Pinterest correlated LLM relevance judgments with human assessments before relying on them.

“requires careful validation to ensure the evaluator LLM's biases don't propagate through the system.”summary, stated
“Pinterest's validation against human judgments shows responsible deployment practices.”summary, stated

Generated journeys can create safety risks on a broad consumer platform.

Root cause: Journey names and recommendations may touch sensitive or inappropriate user contexts.

Resolution: Pinterest routes generated journeys through safety checks before serving.

“The fact that Pinterest explicitly calls out safety checks suggests they're taking responsible deployment seriously”summary, stated

Evaluation

Pinterest evaluated journey quality with human annotations, email feedback, LLM relevance scoring validated against humans, journey-aware notification experiments, and user surveys.

Metrics: LLM-generated 5-level relevance score · 88% higher email click rate · 32% higher push open rate · 23% increase in positive feedback rate · April-May 2025 internal data

3 supporting quotes
“asking it to generate a 5-level relevance score with explanations, they created a scalable evaluation pipeline.”summary, stated
“Compared to existing interest-based notifications, journey-aware notifications showed an 88% higher email click rate and a 32% higher push open rate”summary, stated
“with user surveys revealing a 23% increase in positive feedback rate.”summary, stated

Outcomes

The system was designed for a production platform serving hundreds of millions of users, with full and incremental inference paths to manage freshness and cost.

“The initial deployment of journey-aware notifications demonstrated significant measurable impact.”summary, stated
“they represent a comparison against Pinterest's previous interest-based system rather than against a broader set of baselines or industry benchmarks.”summary, stated