Scaling LLMs for Product Knowledge and Search in E-commerce

doordash · E-commerce · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: e-commerce product knowledge and search

DoorDash needed to scale product knowledge, catalog quality, and personalized multi-intent search as it expanded from food delivery into groceries, alcohol, and retail.

Constraints:

• New verticals involve hundreds of thousands of SKUs across diverse categories.
• Search queries can be multi-intent, multi-entity-type, and location-dependent.
• LLM-powered pipelines must remain high-throughput and low-latency for production use.

Architecture

DoorDash combines traditional ML and LLMs for two major areas: a Product Knowledge Graph with LLM-assisted annotation/attribute extraction and search relevance/personalization. The pipeline starts with manually labeled golden annotations, uses RAG to generate silver annotations, fine-tunes generalized attribute extraction models, detects catalog inconsistencies with LLM prompts over product attributes/images, and uses LLMs for relevance labeling on sparse/tail search queries before deploying smaller distilled/quantized models for online inference.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Product catalog"] --> B["Golden annotations"]
  B --> C["RAG silver annotation generator"]
  C --> D["Attribute extraction model"]
  D --> E["Product Knowledge Graph"]
  A --> F["Catalog inconsistency detector"]
  F --> G["Priority bucket workflow"]
  E --> H["Search relevance system"]
  H --> I["LLM relevance labeling"]
  H --> J["Relevance guardrails"]
  I --> K["Online student models"]

Components

Data flow

Golden annotations seed LLM/RAG generation of silver labels, which fine-tune attribute extraction models and enrich the Product Knowledge Graph. Catalog items are checked by prompts over names, photos, and unit data, then prioritized into P0/P1/P2 fix buckets. Search queries are labeled by LLMs for relevance and personalization models rank results with geo-awareness and guardrails.

Why this architecture

DoorDash uses LLMs where labels, attributes, tail queries, and catalog inconsistencies are expensive for humans, then distills/quantizes models for production latency and cost.

Models used: LLMs · RAG · generalized attribute extraction model · LoRA · QLoRA · Ray · distilled student models

Scale

• DoorDash's new verticals require handling hundreds of thousands of SKUs.
  “Handling hundreds of thousands of SKUs across diverse categories”article, stated
• LLM annotation reduces model-training cycle time from weeks to days.
  “enabling us to train models in days instead of weeks.”article, stated
• Catalog inconsistencies are prioritized into P0, P1, and P2 buckets.
  “classifies the issue into priority buckets — P0, P1, or P2 — based on the severity and urgency of the fix.”article, stated

Alternatives considered (roads not taken)

Human-only annotation for product attributes

Rejected because: Manual labeling is expensive and slow; LLM-assisted silver annotations scale faster.

Simple keyword search

Rejected because: DoorDash search needs multi-intent, multi-entity, geo-aware and personalized relevance.

Let personalization dominate ranking

Rejected because: Over-personalization can show repetitive or irrelevant items and obscure search intent.

Patterns

synthetic-data-generation (data)

DoorDash turns a small golden-label set into many LLM/RAG-generated silver annotations.

rag-retrieval (retrieval)

RAG generates silver annotations and injects external/product knowledge for domain-specific extraction.

model-distillation (model-training)

Distillation and quantization create smaller student models for online inference.

safety-policy-enforcement (ranking)

Relevance guardrails constrain personalization so it does not override user intent.

Tradeoffs

Annotation strategy

Chose golden-plus-silver LLM-assisted annotations over outsourced human annotation only.

Rationale: LLM assistance reduces cost and cycle time while still grounding categories in a small manually labeled set.

Search personalization

Chose personalization with relevance guardrails over unbounded preference-driven ranking.

Rationale: The result must remain relevant to the actual search query.

Online serving model

Chose distilled and quantized student models over large LLMs directly in all online paths.

Rationale: High-throughput low-latency production requires smaller efficient models.

Gotchas (what bit them)

Cold-start categories lack enough high-quality labels.

Root cause: Human annotation for new product categories is slow and expensive.

Resolution: Seed with golden annotations and generate silver annotations with RAG/LLMs.

Catalog errors damage customer trust.

Root cause: Product titles, photos, and units can disagree at SKU scale.

Resolution: Use LLM prompts to classify discrepancies and prioritize P0/P1/P2 fixes.

Tail search queries have sparse and noisy engagement data.

Root cause: Rare queries do not produce enough reliable behavioral labels.

Resolution: Use LLM-generated relevance labels to improve training data quality.

Over-personalization can harm relevance.

Root cause: User preferences can bias ranking away from the actual query intent.

Resolution: Add relevance guardrails around personalization.

Evaluation

DoorDash evaluates product knowledge and search through annotation quality, catalog discrepancy priority, relevance labeling, engagement-derived search signals, guardrail behavior, latency, and production throughput.

Metrics: annotation quality · catalog priority buckets · relevance labels · engagement signals · latency · throughput · model development speed

Outcomes

• DoorDash uses LLM-assisted annotations to train models in days instead of weeks.
• DoorDash enriches its Product Knowledge Graph with extracted product attributes.
• LLMs classify catalog inconsistencies into P0/P1/P2 priority buckets.
• LLM relevance labels improve search training data for sparse tail queries.

The case describes architecture and methods across DoorDash new verticals, with less public quantitative outcome data beyond SKU scale and annotation-time improvement.