Large Language Models for Search Relevance at Scale

pinterest · Tech · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Search relevance ranking at global scale

Pinterest needed better semantic relevance prediction for search across billions of pins, billions of monthly searches, dozens of languages, and many countries. Cross-encoder LLMs improved relevance but were too expensive to serve directly in the high-throughput search ranking pipeline.

Constraints:

• Pinterest handles over 6 billion searches per month
• search spans billions of pins
• the product supports over 45 languages and more than 100 countries
• search relevance is measured on a five-point relevance scale
• the online model must sit late in the ranking pipeline before blending
• human-labeled relevance data is limited and skewed toward specific markets such as the US
• pin representations must be reusable and efficient at production traffic scale

Architecture

Pinterest first fine-tunes multilingual open-source LLM cross-encoders on internal human-labeled relevance data. The teacher concatenates query and pin text, including title, description, visual-language-model synthetic image captions, board titles, and high-engagement search queries, then predicts a five-level relevance score. To productionize it, Pinterest uses semi-supervised distillation: daily search logs are sampled across domains, languages, countries, and fresh pins; the LLM teacher labels them; and a bi-encoder student learns five-scale soft relevance scores. Pin embeddings are inferred offline on billions of pins and refreshed only when inputs meaningfully change, while query embeddings run online in a few milliseconds with about 85% cache hit rate. The resulting relevance-tuned embeddings are reused across related pins, home feed, boards, and other Pinterest surfaces.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Search_query["Search query"] --> Query_understanding["Query understanding"]
  Query_understanding --> Retrieval_candidate_generators["Retrieval candidate generators"]
  Retrieval_candidate_generators --> Early_ranking["Early ranking"]
  Early_ranking --> Production_relevance_student["Production relevance student"]
  Pin_text_features["Pin text features"] --> LLM_cross_encoder_teacher["LLM cross-encoder teacher"]
  Human_relevance_labels["Human relevance labels"] --> LLM_cross_encoder_teacher
  Daily_search_log_sampler["Daily search log sampler"] --> LLM_cross_encoder_teacher
  LLM_cross_encoder_teacher --> Teacher_soft_labels["Teacher soft labels"]
  Teacher_soft_labels --> Production_relevance_student
  Pin_text_features --> Offline_pin_embedding["Offline pin embedding"]
  Offline_pin_embedding --> Production_relevance_student
  Search_query --> Online_query_embedding["Online query embedding"]
  Online_query_embedding --> Query_cache["Query cache"]
  Online_query_embedding --> Production_relevance_student
  BM25_and_graph_features["BM25 and graph features"] --> Production_relevance_student
  Production_relevance_student --> Blending_stage["Blending stage"]
  Offline_pin_embedding --> Reusable_semantic_embeddings["Reusable semantic embeddings"]
  Online_query_embedding --> Reusable_semantic_embeddings

Components

Data flow

A user query enters Pinterest search and candidate generators retrieve pins. At the reranking stage, the production relevance student predicts five-scale relevance scores before blending. The teacher path builds training labels by feeding query+pin text features into a multilingual LLM cross-encoder. Daily search logs provide fresh unlabeled examples; the teacher produces soft relevance labels at 100x scale. The student path embeds pins offline from the same text features, computes query embeddings online, combines bi-encoder embeddings with existing features such as Sage, GraphSage, OmniS, and BM25 statistics, and serves relevance predictions online.

Why this architecture

Pinterest uses a cross-encoder LLM teacher because it can model rich query-pin interactions and improve relevance prediction, but serves a bi-encoder student because pin embeddings can be inferred offline, query embeddings are short and cacheable, and the production model can scale to Pinterest traffic. VLM captions and user-action features expand image and engagement context, while multilingual teacher labels let a limited human-label set generalize across markets.

Models used: fine-tuned open-source LLMs · 8B Llama model · multilingual BERT-based model · SearchSage · bi-encoder student model · visual language model captions · GraphSage · OmniS · BM25

Scale

• Pinterest handles over 6 billion searches each month.
  “Every month we handled over six billion searches with billions of pins to search from.”transcript, stated
• Pinterest search covers billions of pins.
  “over six billion searches with billions of pins to search from.”transcript, stated
• Pinterest search supports over 45 languages.
  “We support over 45 languages and reaching pingers in more than 100 countries.”transcript, stated
• Pinterest search reaches more than 100 countries.
  “We support over 45 languages and reaching pingers in more than 100 countries.”transcript, stated
• The LLM teacher labels data at roughly 100x scale for student training.
  “we label using the teacher and we scale it up pretty much 100x”transcript, stated
• The online query embedding cache hit rate is about 85%.
  “we also cache this u because search queries get repeated a lot and we get around an 85% cash hit rate.”transcript, stated

Alternatives considered (roads not taken)

Serve the cross-encoder LLM teacher directly online

Rejected because: The production system uses a distilled bi-encoder student because cross-encoder interaction is too expensive for online scale.

Train only on human-labeled relevance data

Rejected because: Human labels are small and market-limited, so the team samples daily logs and uses the teacher to scale labels across domains/languages/countries.

Use only text title and description features

Rejected because: VLM captions and user engagement features add useful content annotations for image-heavy pins.

Use separate models for each language

Rejected because: Pinterest used the same multilingual model across languages to transfer improvements internationally.

Patterns

fine-tuning-adaptation (model-training)

Pinterest fine-tunes multilingual open-source LLMs on internal relevance data to adapt them to Pinterest content.

retrieval-ranking (ranking)

The distilled student model predicts five-scale search relevance scores at the reranking stage before blending.

synthetic-data-generation (training-data)

The LLM teacher produces soft labels over sampled daily logs to scale training data roughly 100x across markets.

batch-offline-processing (serving)

Pin embeddings are inferred offline on billions of pins and refreshed only when input features meaningfully change.

cost-aware-routing (serving)

Pinterest distills the expensive teacher into a scalable student, caches query embeddings, and separates offline pin inference from online query inference.

Tradeoffs

Use a cross-encoder LLM as the teacher.

Chose query and pin concatenated into one LLM input over independent query/pin embeddings for the teacher.

Rationale: The cross-encoder better captures query-pin interactions for relevance labeling.

Serve a bi-encoder student online.

Chose offline pin embeddings plus online query embeddings over serving the cross-encoder directly.

Rationale: The bi-encoder scales better because expensive pin embeddings can be precomputed and query embeddings are short/cacheable.

Use semi-supervised teacher labeling.

Chose daily logs labeled by LLM teacher over expanding human labels manually.

Rationale: Teacher labels scale training data across countries, languages, domains, trending queries, and fresh pins.

Use one multilingual model across languages.

Chose same multilingual LM for all languages over separate language-specific models.

Rationale: The multilingual model transfers relevance improvements to non-US/international segments.

Gotchas (what bit them)

Cross-encoder relevance quality does not automatically make a production-search model.

Root cause: Cross-encoders capture query-pin interaction but are too expensive for Pinterest traffic.

Resolution: Use the cross-encoder as teacher and distill into a bi-encoder student with offline and cached embeddings.

Human relevance labels may not cover global markets and fresh content.

Root cause: Most human labels for the experiment came from the US while Pinterest search is multilingual and global.

Resolution: Sample daily search logs and let the multilingual LLM teacher label fresh data across domains, languages, and countries.

Image-heavy search needs richer textual representations than title and description.

Root cause: Pins are visual content, and title/description alone may miss the image semantics and user-curated context.

Resolution: Add VLM-generated captions, board titles, and high-engagement search queries as content annotations.

Online pin embedding would be too expensive at search scale.

Root cause: There are billions of pins, but most pin text features do not change on every query.

Resolution: Infer pin embeddings offline and only refresh changed inputs; infer short query embeddings online and cache repeated queries.

Evaluation

Pinterest evaluates the teacher against existing SearchSage and multilingual BERT baselines, performs ablation studies over text features, measures student online relevance and engagement metrics across US/Germany/France/non-US segments, and checks whether relevance-tuned embeddings improve other surfaces.

Metrics: five-level relevance prediction · 12% improvement over multilingual BERT-based model · 20% improvement over SearchSage embedding model · NDCG · precision@8 · search fulfillment · query embedding latency in a few milliseconds · 85% query cache hit rate

Outcomes

• The 8B Llama teacher improved relevance prediction by 12% over a multilingual BERT-based model.
• The same teacher improved relevance prediction by 20% over the SearchSage embedding baseline.
• The distilled student model delivered international relevance gains even with predominantly US starting labels.
• The query embedding path achieved a few-millisecond latency with around 85% cache hit rate.
• The relevance-tuned embeddings became reusable semantic representations for related pins, home feed, boards, and other surfaces.

Pinterest search handles over 6 billion monthly searches, billions of pins, over 45 languages, and more than 100 countries. The production student uses offline inference over billions of pins and online cached query embeddings to scale.