Large Language Models for Search Relevance via Knowledge Distillation

pinterest · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Search relevance ranking for a visual discovery platform

Pinterest needed higher-quality query-to-Pin relevance predictions for search, but direct cross-encoder LLM serving was too expensive for real-time Pinterest Search. The system also had to represent multimodal Pins as text, use limited human-labeled relevance data efficiently, and generalize beyond US-labeled examples.

Constraints:

• search relevance is evaluated on a five-level guideline
• Pins include images, videos, links, user metadata, board context, and historical engagement signals
• cross-encoder LLM inference is constrained by real-time search latency and cost
• human-labeled relevance data is limited relative to the daily search log volume
• the production model needs to work across languages and countries not covered by the human-labeled set
• the online relevance score must be usable alongside engagement predictions in final ranking

Architecture

Pinterest fine-tuned cross-encoder language models as relevance teachers over query and Pin text, then distilled those teachers into a lightweight online student relevance model. Pin text representations combine titles, descriptions, BLIP-generated synthetic image captions, high-engagement query tokens, board titles, and linked-page metadata. The teacher labels a daily logged search engagement and impression dataset with billions of rows, and the student trains on those teacher labels using query, Pin, and query-Pin interaction features. Online, the student produces relevance scores that are combined with engagement predictions for final search ranking.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  human["Human relevance labels"] --> teacher["Cross-encoder relevance teacher"]
  pintext["Pin text representations"] --> teacher
  teacher --> daily["Daily search log labels"]
  daily --> student["Student relevance model"]
  pintext --> student
  qfeat["Query-level features"] --> student
  pfeat["Pin-level features"] --> student
  pairfeat["Query-Pin interaction features"] --> student
  student --> ranking["Final ranking"]
  teacher --> eval["Offline and online evaluation"]
  student --> eval

Components

Data flow

Pinterest starts with human-annotated relevance data to fine-tune cross-encoder teacher models that read the query and enriched Pin text together. The teacher then labels large daily search logs with five-scale relevance labels. Those generated labels train a smaller student model that embeds query-level features, Pin-level embeddings, and query-Pin interaction features. In production, the student predicts relevance scores for ranking, and those scores are used with engagement predictions to form the final search results.

Why this architecture

Pinterest separated model quality from serving cost: a cross-encoder teacher can capture query-Pin interactions and create high-quality labels offline, while a smaller student can be served online with efficient query, Pin, and interaction features. The enriched Pin text representation lets language models reason over visual and user-curated context, and teacher-generated labels expand training coverage beyond human annotation.

Models used: Llama-3-8B · multilingual BERT-base · T5-base · mDeBERTa-V3-base · XLM-RoBERTa-large · SearchSAGE · PinSAGE · BLIP · feed-forward student relevance model

Scale

• The student is trained from teacher labels on daily search engagement and impression data with billions of rows.
  “generate 5-scale relevance labels on a daily logged large search engagement and impression dataset with billions of rows.”article, stated
• Pinterest uses five relevance levels for query-Pin relevance.
  “To measure the relevance between queries and Pins, we use a 5-level guideline”article, stated
• Llama-3-8B improved five-scale accuracy by 12.5% over multilingual BERT-base.
  “Specifically, the Llama-3–8B outperforms the multilingual BERT-base model by 12.5% and the baseline model by 19.7% in terms of 5-scale accuracy.”article, stated
• The online relevance model improved search feed relevance by 2.18% nDCG@20.
  “Our proposed relevance modeling pipeline leads to a +2.18% improvement in search feed relevance, as measured by nDCG@20.”article, stated
• The final reported business result exceeded 1% search-feed relevance improvement and 1.5% search-fulfillment improvement.
  “online A/B experiment, which showed an improvement of >1% in search feed relevance and >1.5% in search fulfillment rates.”article, stated

Alternatives considered (roads not taken)

Serve the cross-encoder LLM relevance classifier directly in Pinterest Search

Rejected because: The article says direct cross-encoder serving is hard to scale because of real-time latency and cost.

Train only on human-annotated relevance data

Rejected because: Teacher-generated labels let Pinterest use initially unlabeled data and expand coverage across languages and new concepts.

Use only internal SearchSAGE embeddings as the language baseline

Rejected because: Language models provided additional five-scale accuracy improvements over the SearchSAGE-only baseline.

Represent Pins only with titles and descriptions

Rejected because: Ablation results showed each added text feature improved relevance prediction.

Patterns

model-distillation (model-serving)

Pinterest uses the LLM cross-encoder as an offline teacher and trains a lightweight online student relevance model from the teacher's five-scale labels.

batch-offline-processing (data-pipeline)

The expensive LLM teacher runs over daily logged search engagement and impression data offline, rather than in the real-time ranking path.

fine-tuning-adaptation (model-training)

Pinterest fine-tunes multiple pretrained language models on internal search relevance training data, including qLoRA-based fine-tuning for larger Llama models.

synthetic-data-generation (data-generation)

The system uses a teacher model to generate relevance labels at large scale and uses BLIP to generate synthetic image captions for visual Pins.

eval-driven-development (evaluation)

Pinterest compares teacher models, text-feature ablations, label-scale increases, human relevance judgements, and online A/B experiments before claiming production impact.

Tradeoffs

How to use LLM-quality relevance predictions in production search

Chose Offline LLM teacher plus online lightweight student over Directly serving the cross-encoder LLM classifier.

Rationale: The teacher captures richer query-Pin relevance, but the student avoids real-time LLM latency and cost.

How to expand relevance training data

Chose Teacher-generated labels over daily search logs over Relying only on limited human-annotated labels.

Rationale: Teacher labeling scales training examples to unlabeled logs and covers languages, countries, seasonality, and new concepts.

How to represent multimedia Pins to language models

Chose A broad set of text features including generated captions and user-curated signals over Pin titles and descriptions alone.

Rationale: The ablation results show relevance performance improving as each additional text feature is added.

How to validate production relevance changes

Chose Offline metrics, human feed evaluation, cross-language judgements, and online A/B experiments over A single offline accuracy metric.

Rationale: Pinterest evaluates modeling decisions offline, measures human-perceived relevance with nDCG@20, checks country/language generalization, and confirms business impact through search fulfillment.

Gotchas (what bit them)

The most accurate cross-encoder architecture was not directly servable in the search path.

Root cause: A cross-encoder jointly evaluates every query-Pin pair, creating real-time latency and cost pressure.

Resolution: Distill the LLM teacher into a lightweight student relevance model with efficient online features.

Human-labeled data did not cover the full language, country, seasonal, and concept space of Pinterest Search.

Root cause: Human relevance annotations were limited, while search contains global languages and newly emerging concepts.

Resolution: Use multilingual LLM teacher labels over daily logged data to expand training coverage.

Visual Pins need reliable text representations before language models can judge relevance.

Root cause: Pins are multimedia objects, not plain text documents.

Resolution: Build a feature set from Pin metadata, generated captions, board titles, engaged queries, and linked-page metadata.

Large model fine-tuning introduced practical memory and efficiency constraints.

Root cause: Fine-tuning larger models such as Llama requires more memory and compute than smaller encoders.

Resolution: Use quantized model weights, qLoRA, gradient checkpointing, and mixed precision to make training feasible.

Evaluation

Pinterest evaluated the teacher models offline with five-scale accuracy and AUROC thresholds, ran feature ablations, tested the effect of increasing teacher-generated label volume, used human relevance judgements for search-feed nDCG@20 and cross-language precision@8, and ran online A/B experiments using search fulfillment rate.

Metrics: five-scale relevance prediction accuracy · AUROC at relevance thresholds 3, 4, and 5 · feature-ablation accuracy · training-label scale comparisons · human relevance nDCG@20 · cross-country precision@8 · search fulfillment rate

Outcomes

• Llama-3-8B outperformed multilingual BERT-base by 12.5% and the SearchSAGE-only baseline by 19.7% on five-scale accuracy.
• Adding richer Pin text features consistently improved relevance prediction.
• The relevance pipeline improved human-evaluated search feed relevance by 2.18% nDCG@20.
• The online A/B experiment reported more than 1% search-feed relevance improvement and more than 1.5% search fulfillment improvement.
• The system generalized to countries and languages without annotated data available during model training.

The system operates over daily logged search engagement and impression data with billions of rows, trains labels at scale through teacher-generated supervision, and serves lightweight relevance predictions for Pinterest Search.