LLM-Powered Style Compatibility Labeling Pipeline for E-Commerce Catalog Curation

wayfair · E-commerce · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce catalog curation

Wayfair needed scalable style-compatibility labels for product pairs so search and recommendation systems could account for cohesive room design rather than relying only on popularity and behavioral signals.

Constraints:

• The catalog contains tens of millions of unique SKUs.
• Style compatibility is subjective and requires design judgment across imagery and textual metadata.
• Human annotation is accurate but slow and costly.
• The current system is an offline labeling and evaluation pipeline, not yet a live production recommendation service.
• Pairwise product compatibility can become very large at full catalog scale.

Architecture

Wayfair built an offline Google Cloud batch pipeline that pulls product images and metadata, constructs structured multimodal prompts for Gemini 2.5 Pro, and stores JSON outputs containing a Yes/No style-compatibility label plus a design-aware rationale. The prompt uses interior-design dimensions, few-shot examples, and rules about same-category and cross-room compatibility. Labels are evaluated against expert judgments and are used to compare recommendation algorithms before future live search and personalization integration.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Source["Product imagery and metadata source"] --> Prompt["Prompt builder"]
  Prompt --> Judge["Gemini style judge"]
  Judge --> Store["Label store"]
  Store --> Eval["Recommendation evaluation"]
  Expert["Expert holdout set"] --> Eval

Components

Data flow

The batch pipeline collects product imagery and descriptive metadata, builds a structured prompt containing design principles and few-shot examples, sends the product pair to Gemini 2.5 Pro, receives a strict JSON label and rationale, stores the result, and evaluates labels against expert holdout judgments and recommendation-algorithm comparisons.

Why this architecture

Wayfair used an offline batch labeling architecture because the immediate need was scalable annotation for evaluation and curation, not live request-time ranking. Gemini 2.5 Pro was chosen after model benchmarks for accuracy and Google Cloud integration, while prompt engineering was preferred over heavier RLHF or DPO because the team needed fast domain-specific iteration.

Models used: Gemini 2.5 Pro · Gemini variants · open-source multimodal models

Scale

• Wayfair maintains tens of millions of unique SKUs.
  “Wayfair maintains a catalog of tens of millions of unique SKUs”article, stated
• The labeling pipeline is intended for a catalog exceeding millions of items.
  “enabling scalable curation for a catalog exceeding millions of items and unlocking significant improvements in search and recommendation quality.”article, stated
• Wayfair manages tens of thousands of product tags.
  “Wayfair manages tens of thousands of product tags, ranging from objective facts like product dimensions, to fuzzier attributes like color”article, stated
• Prompt iteration improved annotation accuracy by 11%.
  “By iterating on prompt design with expert-labeled example pairs, we improved annotation accuracy by 11%.”article, stated

Alternatives considered (roads not taken)

Traditional human annotation

Rejected because: It was accurate but too slow and costly for scalable style-aware personalization.

RLHF or DPO for the initial capability

Rejected because: Prompt engineering gave Wayfair a faster and lower-overhead way to test hypotheses with domain guidance.

Long rule lists in prompts

Rejected because: A small set of crisp, realistic examples worked better for teaching Wayfair's aesthetic taste.

Immediate live recommendation deployment

Rejected because: Wayfair is first using the labels for offline evaluation and plans live search/personalization integration as the next step.

Patterns

structured-output-contract (integration)

Wayfair constrains model outputs to concise JSON labels and rationales so downstream pipelines can reliably consume annotations.

batch-offline-processing (serving)

The system labels product pairs through an offline Google Cloud batch pipeline rather than live request-time serving.

context-engineering (prompting)

Wayfair curates interior-design principles, product metadata, compatibility rules, and few-shot examples into the prompt so the model reasons with the same context human designers use.

eval-driven-development (evaluation)

Model and prompt choices are evaluated against expert-labeled holdout judgments, and the resulting labels become a metric for comparing recommender algorithms.

data-quality-loop (data)

Wayfair plans active learning for uncertain cases and ongoing monitoring for seasonal/category drift in style labels.

Tradeoffs

Use prompt engineering for the initial style-compatibility system

Chose structured prompts with interior-design rules and few-shot examples over higher-effort RLHF or DPO.

Rationale: Prompt engineering let Wayfair test hypotheses and iterate quickly with domain-specific guidance.

Choose Gemini 2.5 Pro

Chose Gemini 2.5 Pro on Google Cloud over other Gemini variants and open-source multimodal models.

Rationale: Benchmarks showed the highest classification accuracy and clean Google Cloud integration.

Use labels first for offline recommender evaluation

Chose style-compatibility labels as an offline metric for comparing algorithms over immediate customer-facing recommendation changes.

Rationale: Offline evaluation provides a consistent metric before live search and personalization integration.

Gotchas (what bit them)

Style compatibility is subjective rather than perfectly objective.

Root cause: Even human experts vary in style judgments, so perfect agreement is not a realistic target.

Resolution: Evaluate against expert holdout labels while explicitly acknowledging natural human variation.

Generic prompting produced weaker style-label accuracy.

Root cause: Vague prompts did not sufficiently encode interior-design criteria or Wayfair's taste.

Resolution: Add design-aware instructions and curated few-shot examples, yielding an 11% absolute agreement gain.

Full-catalog pairwise labeling is not yet proven at production scale.

Root cause: The system has not yet run Gemini 2.5 Pro against the full catalog and is not live in recommendations.

Resolution: Use the current labels for offline recommendation evaluation first, then expand toward full-catalog and live search/personalization integration.

Style preferences drift over time and categories.

Root cause: Color palettes, materials, and style trends change seasonally and by category.

Resolution: Monitor seasonal and category-specific shifts and refresh the labeling approach as trends evolve.

Evaluation

Wayfair benchmarks models and prompts against a holdout set of expert-labeled product-pair judgments, then plans to reuse style-compatibility labels as an offline recommendation-quality metric.

Metrics: agreement with expert judgments · absolute gain in agreement rate · proportion of recommendation pairs labeled compatible

Outcomes

• Prompt iteration with expert-labeled examples improved annotation accuracy by 11%.
• The system produces scalable Yes/No style-compatibility labels and design-aware rationales.
• Labels now support offline evaluation of recommendation algorithms.
• Live search and personalization integration remains future work.

The pipeline targets Wayfair's tens-of-millions-SKU catalog but has not yet run Gemini 2.5 Pro across the full catalog.