Semantic Relevance Evaluation and Enhancement Framework for E-commerce Search

etsy · E-commerce · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce search relevance evaluation and ranking

Etsy needed search relevance models that reflect buyer intent rather than popularity-biased engagement signals, while still operating at marketplace scale and real-time latency.

Constraints:

• Clicks, add-to-carts, and purchases are measurable but can bias search toward already-popular listings.
• Human labeling every query-listing pair is too slow and expensive.
• Large LLM judges are accurate but too costly for recurring large-scale inference.
• Production search needs lightweight real-time inference with minimal added latency.

Architecture

Etsy built a Semantic Relevance Evaluation and Enhancement Framework with human-curated golden labels, an o3 LLM annotator implemented in LangGraph, a fine-tuned Qwen 3 VL 4B teacher for high-throughput annotation, and a lightweight BERT two-tower student embedded into the search stack for filtering, feature enrichment, loss weighting, and relevance boosting. The teacher runs offline recurring evaluation at scale with vLLM, while the student supports real-time search with under-10ms added latency.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Human golden labels"] --> B["LLM annotator"]
  B --> C["Teacher model training data"]
  C --> D["Teacher model"]
  D --> E["Student model"]
  D --> F["Offline relevance evaluation"]
  F --> G["Search team review"]
  E --> H["Search ranking stack"]
  H --> I["Buyer search results"]

Components

Data flow

Human admins create golden relevance labels and guidelines. Etsy aligns an o3 LLM annotator to those labels, uses it to generate large-scale training data, fine-tunes a Qwen teacher model, distills that teacher into a BERT student, uses the teacher for daily offline measurement, and deploys the student into real-time ranking through filtering, features, loss weighting, and boosting.

Why this architecture

Etsy separates the relevance stack into human labels, LLM annotation, teacher modeling, and student inference because each layer solves a different cost-quality-latency problem: humans define relevance, LLMs scale judgments, the teacher supports high-throughput offline evaluation, and the student keeps realtime search fast.

Models used: o3 · Qwen 3 VL 4B · BERT-based two-tower model · vLLM

Scale

• The teacher labels millions of query-listing pairs daily.
  “This teacher model preserves human alignment while enabling us to label millions of query-listing pairs daily”article, stated
• The student adds less than 10ms of latency in real-time search.
  “The resulting model ensures we deliver search results almost as fast as before, with only <10ms additional latency.”article, stated
• Etsy processes millions of query-listing pairs daily with vLLM for offline evaluation.
  “Using vLLM for high-throughput inference, we process millions of query-listing pairs daily at a very low cost”article, stated
• Fully relevant listings increased from 58% to 62% between August and October 2025.
  “the percentage of fully relevant listings (as defined by the relevance categories described earlier) has increased from 58% to 62% between August and October 2025.”article, stated
• The fairness motivation matters because 89% of Etsy sellers are businesses of one.
  “improving semantic relevance can also help surface items from small or new sellers who may not yet have the visibility of more established shops.”article, stated

  “89% of whom](https://storage.googleapis.com/etsy-extfiles-prod/2025_Q1_GLOBAL_Seller%20Census%202024.pdf?ref=news) are businesses of one.”article, stated

Alternatives considered (roads not taken)

Rely on engagement signals alone

Rejected because: Engagement metrics can reward popularity even when listings are not the best semantic match.

Use human judgments for all query-listing pairs

Rejected because: Full human annotation is too slow and expensive for Etsy-scale search evaluation.

Use the LLM annotator directly for recurrent production-scale inference

Rejected because: The LLM annotator is most aligned to golden labels but too costly for recurrent large-scale use.

Use the teacher model directly in real-time search

Rejected because: The teacher is too slow for realtime search, so Etsy distilled it into a lightweight student.

Patterns

llm-as-judge (evaluation)

Etsy uses an o3 LLM annotator to judge semantic relevance after validating it against human golden labels.

model-distillation (model-adaptation)

Etsy distills an expensive LLM annotator into a Qwen teacher and then into a BERT student for real-time search.

eval-driven-development (evaluation)

Teacher-model semantic relevance metrics are reviewed in monitoring and A/B-test rollout decisions.

human-in-the-loop-gate (governance)

Human-curated golden labels and ongoing admin review anchor the LLM relevance stack.

Tradeoffs

Use LLM annotation after human alignment.

Chose human labels plus scalable LLM judgments over human labels for every query-listing pair.

Rationale: Humans define the target, but LLMs make large-scale annotation feasible.

Distill to a student model for serving.

Chose BERT-based two-tower student over teacher model in realtime search.

Rationale: The student sacrifices some model complexity to preserve real-time latency.

Combine semantic relevance with engagement optimization.

Chose semantic filtering/features/boosting in the search stack over replacing production ranking with semantic labels alone.

Rationale: The framework improves relevance through multiple integration points while search remains influenced by multiple factors.

Gotchas (what bit them)

Engagement labels create popularity bias.

Root cause: Search models trained on clicks and purchases reward items that already have visibility.

Resolution: Add semantic relevance labels and models that directly judge query-listing intent match.

Off-the-shelf LLM judges may not match Etsy's marketplace semantics.

Root cause: Etsy's user vocabulary and preferences create domain shift from generic LLM knowledge.

Resolution: Align the LLM annotator with human-curated golden labels and evolving relevance guidelines.

The most accurate model is not the production-serving model.

Root cause: The LLM annotator and teacher are too expensive or slow for recurrent production inference.

Resolution: Use cascaded distillation down to a lightweight realtime student.

Relevance definitions drift with cultural context.

Root cause: The same query can mean different things over time, such as face masks before and after 2020.

Resolution: Maintain evolving relevance guidelines informed by research and annotation feedback.

Evaluation

Etsy evaluates semantic relevance with human golden labels, Macro F1 and class F1 against golden data, daily teacher-model offline inference, row-level disagreement rates, and A/B-test relevance metrics considered before search rollouts.

Metrics: multi-class Macro F1 · individual class F1 · row-level disagreement rate · percentage of fully relevant listings · A/B-test semantic relevance metrics · additional latency

Outcomes

• Semantic relevance is fully deployed in Etsy's search stack.
• Fully relevant listings increased from 58% to 62% between August and October 2025.
• The student model adds less than 10ms latency while supporting realtime search enhancement.
• The framework helps surface relevant items from smaller or newer sellers by reducing popularity bias.

Etsy reports millions of query-listing pairs processed daily for offline relevance evaluation and a realtime student model with less than 10ms added latency.