LLM-Powered Search Evaluation System for Automated Result Quality Assessment

doordash · E-commerce · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Search-result page relevance evaluation

DoorDash needed to evaluate search relevance at the whole-page level across diverse verticals without being bottlenecked by slow, inconsistent, and expensive human annotation.

Constraints:

• Manual relevance annotation could not scale to millions of query-document pairs.
• Human feedback loops took days or weeks, slowing search iteration.
• Human raters interpreted guidelines inconsistently.
• Head queries were overrepresented while tail-query problems were undercovered.
• DoorDash's 2-D search result pages require page-level relevance, not only vertical-list ranking metrics.

Architecture

DoorDash built AutoEval, a human-in-the-loop LLM evaluation system that samples real user queries, constructs structured prompts for query-result tasks, runs base or fine-tuned LLM inference, aggregates judgments into a Whole-Page Relevance score, and audits sampled outputs with external raters and internal experts. Fine-tuned GPT-4o models and expert-labeled golden datasets improve alignment, while WPR supports offline feature evaluation and daily production relevance monitoring.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Sampler["Query sampler"] --> Prompt["Prompt constructor"]
  Prompt --> Judge["LLM judge"]
  Judge --> WPR["WPR aggregator"]
  WPR --> Eval["Offline and production evaluation"]
  Eval --> Audit["Human audit loop"]
  Audit --> Golden["Golden dataset"]
  Golden --> Judge

Components

Data flow

AutoEval samples live queries across intent, frequency, geography, and daypart; converts each query-result pair into a task-specific prompt; sends it to a base or fine-tuned LLM; rolls judgments into WPR page scores; and feeds sampled judgments into human auditing and expert refinement loops.

Why this architecture

DoorDash used LLM evaluation because the old human-labeling process could not match the scale and iteration speed of modern search. WPR was necessary because DoorDash search pages have multiple spatial content blocks, and expert auditing was kept because automated judgments still need calibration against evolving domain standards.

Models used: GPT-4o · fine-tuned GPT-4o · LLMs

Scale

• AutoEval targets automated assessments of millions of relevance judgments per day.
  “Automated assessments of millions of relevance judgments per day.”article, stated
• Manual annotation cycles previously took days or weeks.
  “Slow feedback loops: Human annotation cycles can take days or weeks, slowing iteration speed for search improvements.”article, stated
• AutoEval reduced relevance-judgment turnaround time by 98%.
  “AutoEval has reduced relevance judgment turnaround time by 98% compared to human evaluation”article, stated
• AutoEval unlocked a nine-fold increase in evaluation capacity.
  “unlocking a nine-fold increase in capacity and resolving a major bottleneck in our offline experimentation pipeline.”article, stated
• DoorDash search spans restaurants, retail, grocery, and pharmacy.
  “DoorDash scaled to support diverse verticals, including restaurants, retail, grocery, and pharmacy.”article, stated

Alternatives considered (roads not taken)

Traditional human-driven relevance annotation

Rejected because: It was too slow, costly, inconsistent, and coverage-limited for modern DoorDash search.

"Among the challenges are:

• Scalability constraints: It isn’t feasible to manually assess millions of query-document pairs"

— article, stated

NDCG-style vertical-list evaluation

Rejected because: DoorDash pages contain spatially arranged content blocks, so the metric needed to evaluate whole-page usefulness.

Zero-shot or generic prompting

Rejected because: Task-specific structured prompts with rule logic and domain examples were more consistent and interpretable.

Rely on a single public LLM provider indefinitely

Rejected because: DoorDash plans to route through an internal gateway to compare cost, latency, and accuracy across vendors.

Patterns

eval-driven-development (evaluation)

AutoEval is DoorDash's evaluation layer for offline experiments, daily production relevance monitoring, model benchmarking, and quality improvement loops.

human-in-the-loop-gate (quality)

External raters and internal experts review LLM judgments, investigate low-quality outputs, and feed corrections back into prompts and golden data.

fine-tuning-adaptation (model-adaptation)

DoorDash fine-tunes LLMs on expert-labeled relevance data and justifications for key evaluation categories.

prompt-chaining (prompting)

AutoEval uses chain-of-thought and stepwise fallback logic to mirror evaluator decision processes.

structured-output-contract (judgment-schema)

The LLM returns structured relevance judgments from task-specific prompts that reflect internal rating rubrics.

observability-tracing (monitoring)

WPR and human audits support continuous production relevance monitoring and stability checks.

Tradeoffs

Automate relevance judgments with oversight

Chose LLM-powered evaluation plus human auditing over manual relevance annotation as the primary evaluation path.

Rationale: The hybrid approach improved scale and speed while keeping expert judgment focused on calibration and edge cases.

Measure whole-page relevance

Chose WPR over individual-result-only relevance over NDCG-style vertical-list assumptions.

Rationale: WPR matches the spatial, multi-block search experience DoorDash users actually see.

Fine-tune for high-impact categories

Chose expert-labeled golden data and fine-tuning over prompt-only LLM judging.

Rationale: Fine-tuning improved alignment in store-name, cuisine, and dish/item relevance tasks.

Plan provider abstraction

Chose internal GenAI gateway over direct dependence on OpenAI public API.

Rationale: The gateway would let DoorDash compare vendors across cost, latency, and accuracy without changing downstream systems.

Gotchas (what bit them)

Human raters create slow and inconsistent search feedback.

Root cause: Manual annotation cycles take days or weeks and raters interpret guidelines differently.

Resolution: Use LLM-powered judgments with expert-created rubrics and audit loops.

Head-query bias hides tail-query relevance problems.

Root cause: Annotated datasets overrepresent frequent queries and underrepresent tail queries.

Resolution: Sample across intent, frequency, geography, and daypart and use LLM scale to broaden coverage.

Search-page relevance is not just top-result relevance.

Root cause: DoorDash UI contains stores, dishes, and items arranged spatially with different visual prominence.

Resolution: Use WPR to weight the whole page by content blocks and expected user impact.

Fine-tuned judges still require ongoing calibration.

Root cause: Search domains, edge cases, and quality standards evolve.

Resolution: Keep external audits, internal expert investigation, prompt refinement, new golden data, and ongoing fine-tuning in the loop.

Evaluation

DoorDash evaluates AutoEval through expert-labeled golden datasets, train/eval splits, model benchmarking, external rater audits, internal expert investigation, WPR offline impact checks, daily production monitoring, and offline benchmark comparisons with external raters.

Metrics: 98% relevance judgment turnaround-time reduction · nine-fold capacity increase · fine-tuned GPT-4o outperformed external raters in offline benchmark accuracy · WPR daily production relevance monitoring

Outcomes

• AutoEval reduced relevance-judgment turnaround time by 98%.
• Evaluation capacity increased nine-fold.
• Fine-tuned LLMs matched or outperformed external raters in key relevance tasks.
• Expert raters could focus on guidelines, auditing, and edge cases.
• WPR became a daily relevance-monitoring and offline feature-evaluation metric.

DoorDash reports automated assessment of millions of relevance judgments per day, a 98% turnaround-time reduction, nine-fold capacity increase, and multiple verticals including restaurants, retail, grocery, and pharmacy.