Multimodal LLM-as-a-Judge for Large-Scale Product Retrieval Evaluation

zalando · E-commerce · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce retrieval evaluation

Zalando needed to evaluate product retrieval relevance continuously across languages and query types without relying on slow, costly human annotation for every large-scale search-quality check.

Constraints:

• The platform serves diverse customer queries across multiple languages.
• Human relevance assessments are expensive and slow at large scale.
• Product relevance depends on both text and images.
• The evaluation system needed context-specific annotation guidelines rather than one static rubric.

Architecture

Zalando built a multimodal LLM-as-a-judge framework that generates query-specific annotation guidelines, inspects product text and images, and scores retrieval relevance at production-monitoring scale.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  retrieval-evaluation-examples["Retrieval Evaluation Examples"]
  guideline-generator["Guideline Generator"]
  multimodal-judge["Multimodal Judge"]
  production-monitoring["Production Monitoring"]
  retrieval-evaluation-examples --> guideline-generator
  guideline-generator --> multimodal-judge
  multimodal-judge --> production-monitoring

Components

Data flow

Search examples are routed into the framework, an MLLM generates context-specific guidelines, the judge analyzes product text and images, and relevance judgments feed continuous monitoring of high-frequency search queries.

Why this architecture

The framework substitutes scalable multimodal judging for human-only labeling while keeping task context in the generated annotation guidelines.

Models used: Multimodal Large Language Models

Scale

• The framework was evaluated on 20,000 examples.
  “Evaluated on 20,000 examples”summary, stated
• The approach was up to 1,000 times cheaper than human annotation.
  “up to 1,000 times cheaper and significantly faster (20 minutes versus weeks for humans)”summary, stated
• The automated run took about 20 minutes compared with weeks for human annotators.
  “20 minutes versus weeks for humans”summary, stated

Alternatives considered (roads not taken)

Human-only relevance assessment

Rejected because: It is too slow and costly for continuous large-scale retrieval monitoring.

Text-only evaluation

Rejected because: Fashion-product relevance depends on product images as well as text.

Patterns

llm-as-judge (evaluation)

Multimodal LLMs score product-retrieval relevance as an automated evaluator.

eval-driven-development (evaluation)

The system enables continuous monitoring of frequent queries to identify where retrieval needs improvement.

retrieval-ranking (retrieval)

The judge evaluates product retrieval relevance so ranking/search systems can be monitored and improved.

Tradeoffs

Automate relevance assessment

Chose Multimodal LLM-as-a-judge over Continuous human-only annotation.

Rationale: Automation made continuous evaluation economically and operationally feasible.

Use multimodal evidence for relevance

Chose Text and image analysis over Text-only query/product evaluation.

Rationale: Fashion retrieval relevance often depends on visual product attributes.

Gotchas (what bit them)

Static evaluation rubrics miss query-specific relevance criteria.

Root cause: Product relevance changes with query intent and product modality.

Resolution: Generate context-specific annotation guidelines for each judgment.

Human labeling cadence cannot keep up with high-frequency query monitoring.

Root cause: Manual annotation takes weeks and substantial resources.

Resolution: Use MLLM judging for fast continuous monitoring, with human comparability checked on a labeled sample.

Evaluation

Zalando compared MLLM judgments with human annotations on a 20,000-example evaluation set and measured speed and cost.

Metrics: Accuracy versus human annotators · Evaluation cost · Evaluation runtime

Outcomes

• The MLLM judge achieved human-comparable accuracy on the reported evaluation.
• Evaluation cost dropped by up to 1,000x.
• Evaluation time dropped from weeks to about 20 minutes.

The framework was tested on 20,000 examples and intended for continuous production monitoring of high-frequency queries.