LLM-Powered Relevance Assessment for Search Results

pinterest · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Search relevance measurement for online experiments

Pinterest Search needed cheaper and faster relevance labels so A/B experiments could detect small ranking-quality changes and heterogeneous treatment effects rather than only large topline movements.

Constraints:

• Human relevance annotations had low availability and high marginal cost.
• Previous sample sizes only detected large topline metric movements.
• The relevance model had to support multilingual queries and Pins.
• Model choice had to balance accuracy against inference time and cost.
• Experiment metrics needed validation at query-level and paired-difference levels, not only individual-label accuracy.

Architecture

Pinterest fine-tuned multilingual open-source language models as cross-encoder relevance classifiers over query-Pin pairs. The selected XLM-RoBERTa-large model generates five-level relevance labels for the top 25 search results in paired A/B experiment samples. LLM labeling enables larger stratified query samples, sDCG@25 computation, heterogeneous-effect analysis, and lower minimum detectable effects while validation compares LLM labels and derived metrics against human annotations.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Labels["Human relevance labels"] --> Model["Cross-encoder relevance model"]
  PinText["Pin text representation"] --> Model
  Model --> Labeling["LLM labeling pipeline"]
  Labeling --> Metrics["Experiment metrics pipeline"]
  Sampling["Stratified sampling design"] --> Metrics
  Metrics --> Validation["Validation suite"]

Components

Data flow

Human-annotated query-Pin pairs train a cross-encoder relevance model. The model consumes query text and a rich Pin representation, emits five-dimensional relevance scores, labels top-K experiment results, computes query-level sDCG@25, aggregates topline metrics, and analyzes heterogeneous effects by popularity and interest segments with false-discovery-rate control.

Why this architecture

Pinterest chose a fine-tuned multilingual cross-encoder because relevance labels needed high-quality query-Pin judgment for offline experiment measurement, while reduced labeling cost made larger stratified samples feasible. XLM-RoBERTa-large was selected over Llama-3-8B because it delivered sufficient quality at far lower inference time and cost.

Models used: XLM-RoBERTa-large · Llama-3-8B · multilingual BERT-base · T5-base · mDeBERTa-V3-base · DistilBERT · BLIP

Scale

• The production model labels 150,000 rows within 30 minutes on a single A10G GPU.
  “Inference runs on a single A10G GPU, allowing us to label 150,000 rows within 30 minutes.”article, stated
• Pinterest reduced relevance-measurement MDE from roughly 1.3%-1.5% to at most 0.25%.
  “Before the introduction of LLM labeling, relevance measurement had large MDEs (e.g. 1.3%-1.5%).”article, stated
• LLM-generated labels achieved 73.7% exact match with human labels and 91.7% within one point.
  “LLM-generated labels and human labels yield an exact match rate of 73.7%, with 91.7% of ratings deviating at most by 1 point.”article, stated
• The production relevance metric uses K=25 search results per query.
  “We use K=25 throughout our evaluation.”article, stated
• Non-English validation focused on France and Germany.
  “For this analysis, we focus on France (FR) and Germany (DE) markets.”article, stated

Alternatives considered (roads not taken)

Human annotations for all relevance measurement

Rejected because: Human labels had low availability, high marginal cost, and forced sample designs that could only detect large effects.

Simple random sampling with human-label economics

Rejected because: The LLM cost structure made stratified query sampling and larger samples practical, reducing MDE.

Llama-3-8B as the production relevance model

Rejected because: It offered only slight accuracy improvement while increasing inference time and cost by 6x.

Text-only Pin metadata without visual captions

Rejected because: Pinterest used BLIP-generated captions as part of a richer Pin representation, while leaving raw-image VLMs as future work.

Patterns

fine-tuning-adaptation (model-adaptation)

Pinterest fine-tunes multilingual open-source LLMs on human relevance labels for five-level query-Pin relevance classification.

eval-driven-development (evaluation)

The entire system is a production evaluation layer for A/B tests, with LLM labels validated against human labels and experiment metrics.

model-routing (model-selection)

Pinterest evaluated multiple candidate models and selected XLM-RoBERTa-large for the best quality/efficiency balance rather than maximum raw accuracy.

latency-optimization (inference)

The production model choice keeps labeling throughput practical: 150,000 rows in 30 minutes on one A10G GPU.

Tradeoffs

Use fine-tuned LLM relevance labels

Chose model-generated labels trained on human annotations over ongoing human annotation as the primary measurement source.

Rationale: LLM labels cut cost and time enough to enable larger samples and better experiment sensitivity.

Use a cross-encoder relevance model

Chose joint query-Pin scoring over cheaper independent representations.

Rationale: The source presents the cross-encoder as the relevance-prediction architecture used for query-Pin judgment.

Prioritize XLM-RoBERTa-large for production

Chose lightweight high-quality multilingual model over Llama-3-8B's slightly higher accuracy.

Rationale: XLM-RoBERTa-large balanced prediction quality with inference efficiency.

Scale and stratify experiment samples

Chose larger stratified query sampling over simple random sampling under human-label constraints.

Rationale: Stratification reduced variance and made heterogeneous treatment effects measurable.

Gotchas (what bit them)

Human labels can make experiment metrics too insensitive.

Root cause: Low label availability and high marginal cost force small samples and large MDEs.

Resolution: Use fine-tuned LLM labeling to scale sample sizes and enable stratified sampling.

The highest-accuracy model may be operationally wrong.

Root cause: Llama-3-8B had slightly better accuracy but materially higher inference time and cost.

Resolution: Select XLM-RoBERTa-large as the production model.

Individual-label accuracy is not enough to validate experiment measurement.

Root cause: A/B tests depend on query-level metrics and paired differences, not just Pin-level exact match.

Resolution: Validate query-level sDCG@K correlations, error distributions, and paired-difference errors.

Multilingual relevance assessment needs separate validation.

Root cause: Most training query-Pin pairs were English, creating a possible performance gap for non-English queries.

Resolution: Validate France and Germany query-level metric alignment before relying on non-English LLM labels.

Evaluation

Pinterest validated LLM labeling through Pin-level exact match and within-one agreement, query-level Kendall and Spearman correlations, sDCG@25 error distributions, paired-difference errors for A/B tests, popularity segments, and non-English market validation for France and Germany.

Metrics: 73.7% exact match with human labels · 91.7% within one relevance point · Kendall's tau > 0.5 for US popularity segments · Spearman's rho > 0.65 for US popularity segments · overall query-level error below 0.01 · MDE reduced to <=0.25% · 150,000 rows in 30 minutes on one A10G GPU

Outcomes

• Pinterest deployed LLM-based relevance assessment for Search A/B experiments.
• The system reduced manual annotation costs and turnaround time.
• MDE dropped by approximately an order of magnitude to at most 0.25%.
• The pipeline enabled heterogeneous treatment-effect analysis by query popularity and interest.
• The selected model can label 150,000 rows in 30 minutes on a single A10G GPU.

The case reports 150,000 labels in 30 minutes on one A10G GPU, K=25 results per query, exact and within-one agreement metrics, and MDE reduction from 1.3%-1.5% to <=0.25%.