Large-Scale Personalization and Product Knowledge Graph Enhancement Through LLM Integration

doordash · E-commerce · year n/a · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce personalization and product knowledge graph automation

DoorDash needed to scale product understanding and personalization as it expanded from restaurant menus into grocery, convenience, alcohol, flowers, pets, retail, and electronics. The company had to convert noisy merchant data into standardized SKUs, build product attributes and relationships, support cold-start personalization for new customers and new products, avoid dangerous hallucinations for high-stakes attributes such as allergens, and make recommendations across a huge and expanding catalog.

Constraints:

• restaurant menus may have about 100 items while retail merchants can have around 100,000 items
• the marketplace includes merchants, consumers, and dashers with ML needs across nodes and edges
• product knowledge must include flavor, size, dietary attributes, store availability, inventory, substitutions, and ontology relationships
• merchant data is noisy, incomplete, and often cryptic
• categorization spans about 3,000 categories
• DoorDash processes on the order of 100 million SKUs
• allergen and health-related attributes require high precision before production use
• cold-start personalization has little or no engagement history for new customers, products, or verticals

Architecture

DoorDash layers LLMs and generative AI on top of traditional ML. The product knowledge graph starts from merchant data and moves from human spreadsheet processing to fine-tuned LLM attribute extraction, RAG-based categorization, and LLM agents for ambiguous product strings that need internal or external lookup. Scaling uses model cascading: heuristics, in-house fine-tuned models, and frontier/API models only for the remaining hard cases, plus output distillation into smaller models. Personalization uses hierarchical RAG over product taxonomy and catalog nodes, LLM world knowledge for complementary recommendations, mapping back to catalog categories, and LLM-based evaluation with scored relevance and explanations. Future work adds semantic IDs and contextual agents, but the talk says that agentic shopping example is not yet production.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  merchant["Merchant product data"] --> extract["Attribute extraction pipeline"]
  merchant --> agent["LLM catalog agent"]
  extract --> cascade["Model cascade"]
  cascade --> distill["Output distillation pipeline"]
  extract --> kg["Product knowledge graph"]
  categories["Category vector index"] --> agent
  agent --> kg
  kg --> personalize["Personalization models"]
  kg --> rag["Hierarchical RAG recommender"]
  personalize --> rag
  rag --> evals["Evaluation pipeline"]
  evals --> personalize
  semantic["Semantic ID roadmap"] --> rag
  semantic --> future["Future contextual agents"]
  kg --> future

Components

Data flow

Merchant product data enters the catalog pipeline and is standardized into DoorDash SKUs. Fine-tuned LLMs extract attributes; for large category spaces, vector search retrieves candidate categories and an LLM chooses among them; for cryptic product records, an LLM agent can search internal databases, manufacturer websites, or third-party sources before producing product names. The product knowledge graph feeds downstream personalization. For complementary recommendations, the system starts from a terminal product node, retrieves relevant product types, uses an LLM to rank likely complements, maps results back to the catalog, recursively unfolds child nodes, and finally retrieves concrete items. LLM judges or councils score recommendation relevance and feed explanations back into model improvement.

Why this architecture

DoorDash keeps traditional ML as the foundation because engagement and preference models still drive personalization. LLMs add world knowledge, language understanding, and reasoning where engagement is sparse, catalog data is noisy, or category spaces are too large for static rules. RAG reduces the 3,000-category context problem before the LLM makes a decision. Agents are reserved for ambiguous records where the model needs external evidence. Cascading and distillation make production cost and latency manageable at roughly 100-million-SKU scale.

Models used: GPT-4o · GPT-4o mini · Llama · LLM agents · RAG · Faiss · GPT embeddings · open-source embeddings · LLM-as-judge · council of LLMs · traditional two-tower embeddings

Scale

• Restaurant menus have about 100 items, while a Best Buy-like retailer may have around 100,000 items.
  “a restaurant menu probably has 100 items, but if you think about a Best Buy, they might haveund,000 items”transcript, stated
• DoorDash is expanding beyond restaurants into grocery, convenience, alcohol, flowers, pets, retail, and electronics.
  “new verticals. that includes grocery, convenience, alcohol, flowers, pet, retail, you know, talk about it electronics”transcript, stated
• The categorization problem covers about 3,000 categories.
  “Well, there's about 3,000 categories.”transcript, stated
• The SKU processing scale is on the order of 100 million SKUs.
  “we have millions and millions order of 100 million SKs that we need to process.”transcript, stated
• The model cascade handles about 88% of SKUs before large LLM calls.
  “if 88% of our skewers are handled before hitting the large LLM”transcript, stated
• The DoorDash marketplace has merchant, consumer, and Dasher sides.
  “there's the merchant, there's the the consumer who is placing the order, and then there's the Dasher”transcript, stated

Alternatives considered (roads not taken)

Keep product knowledge graph construction human-driven

Rejected because: Manual spreadsheet-based processing was slow, expensive, and error-prone.

Use prompt tuning alone for attribute extraction

Rejected because: DoorDash says prompt tuning did not work very well for domain-specific attributes, so fine-tuning was needed.

Put all 3,000 categories in the LLM context window

Rejected because: Large context increased hallucination, slowed responses by seconds, and produced poorer metrics.

Send every SKU to a frontier/API model

Rejected because: DoorDash uses heuristics and in-house fine-tuned models first so only a small fraction reaches expensive large models.

Use RAG and fine-tuning as mutually exclusive choices

Rejected because: The speaker says they are complementary, with RAG reducing a larger set before a fine-tuned LLM performs ranking.

Patterns

fine-tuning-adaptation (model-adaptation)

DoorDash fine-tuned LLMs on e-commerce data for domain-specific attribute extraction and precision-sensitive catalog tasks.

rag-retrieval (retrieval)

DoorDash uses RAG to reduce large category/product spaces before LLM ranking and to generate complementary recommendations from product taxonomy.

tool-use-orchestration (agentic-catalog)

LLM agents automate the human lookup workflow by searching internal databases, manufacturer websites, or third-party sources for ambiguous merchant data.

cost-aware-routing (serving)

DoorDash cascades from heuristics to smaller fine-tuned models to frontier/API models so most SKUs avoid expensive large-model calls.

model-distillation (serving)

DoorDash distills larger model outputs into smaller models to improve production throughput, latency, and cost.

llm-as-judge (evaluation)

DoorDash uses a stronger LLM or council of LLMs with chain-of-thought reasoning to score recommendation relevance.

data-quality-loop (catalog-quality)

Evaluation explanations and generated labels feed back into traditional ML, RAG recommendations, search relevance, and model refinements.

Tradeoffs

Product attribute extraction approach

Chose Fine-tuned smaller models after GPT-4-class baselines over Relying only on larger prompted models.

Rationale: Fine-tuning improved task performance, and smaller models lowered cost and increased speed while maintaining useful performance.

Category lookup strategy

Chose RAG over a category vector index over Long-context prompt with all categories.

Rationale: Retrieval reduced context size, improved top-K metrics, reduced hallucination, and sped responses.

SKU processing cost strategy

Chose Model cascade over Single frontier model for every SKU.

Rationale: Heuristics and in-house fine-tuned models handle most SKUs before expensive frontier models, reducing latency and cost.

Cold-start personalization strategy

Chose Blend traditional ML with LLM world knowledge over Wait for user/product engagement history.

Rationale: Traditional engagement models fail for new customers or products; LLMs can supply prior world knowledge that is then mapped back to the catalog.

Gotchas (what bit them)

Product-catalog automation is high stakes when attributes affect health.

Root cause: Attributes such as allergen-free status need high precision before production exposure.

Resolution: Fine-tune LLMs on e-commerce data and use strong evaluation to meet the required accuracy bar.

Long-context category prompts degrade quality and latency.

Root cause: Supplying thousands of categories to an LLM increases hallucination, slows response time, and worsens metrics.

Resolution: Use vector search to retrieve top candidate categories, then ask the LLM to decide among a smaller set.

Cryptic merchant data can defeat both direct LLM and RAG approaches.

Root cause: Abbreviated product strings may omit facts needed for exact product identification.

Resolution: Use LLM agents with internal/external search tools to mimic the previous human lookup workflow.

Agents introduce hallucination risk even when they improve lookup.

Root cause: Tool-calling agents reason over external information and synthesize outputs, so unsupported names can still appear.

Resolution: Use strong evaluations, prompt tuning, reasoning-process tuning, and validation after results are produced.

LLM world knowledge still lacks DoorDash's full catalog context.

Root cause: The catalog space is huge and cannot be fully placed into the model context.

Resolution: Map generated recommendations back to catalog nodes and invest in semantic IDs to simplify RAG and catalog mapping.

Evaluation

DoorDash evaluates catalog and personalization systems with task-performance comparisons, strong evaluations for agents, top-K category metrics, LLM or council-of-LLMs judging with chain-of-thought reasoning, three-level relevance scores, metrics pipelines, and explanation feedback loops.

Metrics: attribute extraction task performance · top-K candidate category metrics · hallucination reduction · latency reduction · cost reduction · SKU coverage before frontier-model calls · LLM judge score 3/2/1 relevance · aggregate metrics pipeline performance · recommendation explanation feedback

Outcomes

• DoorDash shifted product knowledge graph construction from slow manual processing toward LLM-driven automation.
• Fine-tuned smaller models reduced cost, increased speed, and kept useful attribute-extraction performance.
• RAG over category vectors improved categorization metrics and response speed compared with sending all categories to the LLM.
• LLM agents improved ambiguous product-name resolution by using external/internal search tools.
• Model cascading handles about 88% of SKUs before large LLM calls, reducing latency and cost.
• Hierarchical RAG enables complementary recommendations and cold-start personalization by combining product taxonomy, catalog retrieval, and LLM world knowledge.
• Semantic IDs and contextual shopping agents are identified as future investments, not current production in the tornado example.

DoorDash describes expansion from ~100-item restaurant menus to ~100,000-item retailers, a three-sided marketplace of merchants/consumers/dashers, about 3,000 categories, order-of-100-million SKUs, 88% SKU handling before large-LLM calls in the cascade, product spaces that expand deeply and broadly, and future agentic workflows over weather, inventory, delivery, long/short-term memory, and cart-building.