Building a Food Delivery Product Knowledge Graph with LLMs

doordash · E-commerce · year n/a · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Retail catalog enrichment and product knowledge graphs

DoorDash needed to enrich and standardize highly heterogeneous merchant SKU data so new-verticals products could support search, fulfillment, personalization, substitutions, ads, and global catalog entity resolution.

Constraints:

Architecture

DoorDash used LLMs as enrichment engines around its retail catalog and product knowledge graph. The system routes low-confidence brand cases from an in-house classifier to an LLM, uses another LLM with internal knowledge-graph retrieval to detect duplicates, adds new brands back into the knowledge graph, and retrains the classifier. For other attributes, it combines string matching, LLM reasoning, LLM agents with web search, embeddings-based RAG over golden annotations, GPT-4 label generation, and fine-tuning for scalable inference.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  raw-merchant-sku-data[Raw merchant SKU data] --> retail-catalog[Retail catalog]
  raw-merchant-sku-data --> in-house-brand-classifier[In-house brand classifier]
  in-house-brand-classifier --> brand-extraction-llm[Brand extraction LLM]
  brand-extraction-llm --> duplicate-detection-llm[Duplicate detection LLM]
  duplicate-detection-llm --> product-knowledge-graph[Product knowledge graph]
  product-knowledge-graph --> in-house-brand-classifier
  raw-merchant-sku-data --> organic-label-pipeline[Organic label pipeline]
  organic-label-pipeline --> external-search-agent[External search agent]
  organic-label-pipeline --> retail-catalog
  raw-merchant-sku-data --> embedding-retriever[Embedding retriever]
  embedding-retriever --> golden-annotation-set[Golden annotation set]
  embedding-retriever --> gpt-4-label-generator[GPT-4 label generator]
  gpt-4-label-generator --> fine-tuned-attribute-model[Fine-tuned attribute model]
  fine-tuned-attribute-model --> retail-catalog
  retail-catalog --> downstream-ml-models[Downstream ML models]
  product-knowledge-graph --> downstream-ml-models
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Raw merchant SKU data Unstandardized product names, descriptions, images, and merchant-provided attributes. Retail catalog, In-house brand classifier, Embedding retriever
Retail catalog Centralized dataset of product attributes for new-verticals merchant products. Raw merchant SKU data, Product knowledge graph, Downstream ML models
In-house brand classifier Existing classifier that tags SKUs to known brands when confidence is high. Raw merchant SKU data, Brand extraction LLM, Product knowledge graph
Brand extraction LLM Extracts candidate brands from SKUs the classifier cannot tag confidently. In-house brand classifier, Duplicate detection LLM
Duplicate detection LLM Uses internal graph examples to decide whether an extracted brand is a duplicate entity. Brand extraction LLM, Product knowledge graph
Product knowledge graph Stores brand hierarchy, product entities, and extracted attributes for downstream catalog use. Retail catalog, Duplicate detection LLM, In-house brand classifier, Downstream ML models
Organic label pipeline Combines string matching, LLM reasoning, and LLM agents for organic product tagging. Raw merchant SKU data, External search agent, Retail catalog
External search agent Searches product information online and passes results to another LLM for organic reasoning. Organic label pipeline
Embedding retriever Finds similar SKUs from golden annotations using OpenAI embeddings and approximate nearest neighbors. Raw merchant SKU data, Golden annotation set, GPT-4 label generator
Golden annotation set Human-generated reference annotations used as in-context examples. Embedding retriever, GPT-4 label generator
GPT-4 label generator Generates labels for unannotated SKUs from retrieved in-context examples. Embedding retriever, Fine-tuned attribute model
Fine-tuned attribute model Scalable inference model trained from generated annotations. GPT-4 label generator, Retail catalog
Downstream ML models Ranking, personalization, substitution, and ads models that consume enriched attributes. Retail catalog, Product knowledge graph

Data flow

Raw merchant SKU data enters the retail catalog. Existing classifiers tag known brands when confident. Low-confidence SKUs go to an LLM for extraction, then a second LLM retrieves similar brands and example item names from the internal knowledge graph to decide whether a candidate is duplicate. Accepted brands enter the graph and retrain the classifier. For generalized attributes, OpenAI embeddings retrieve similar golden annotations, GPT-4 generates labels, and those generated annotations fine-tune a model for scalable inference.

3 supporting quotes
“1. Unstructured product description is passed to our in-house brand classifier 2. SKUs that cannot be tagged confidently to one of the existing brands are passed to an LLM for brand extraction”article, stated
“SKUs that cannot be tagged confidently to one of the existing brands are passed to an LLM for brand extraction”article, stated
“For each unannotated SKU, we first leverage OpenAI embeddings and the approximate nearest neighbors technique to retrieve the most similar SKUs from our golden annotation set.”article, stated

Why this architecture

DoorDash used LLMs to bypass the cold start of building specialized extractors from scratch, but did not rely on one generic prompt for everything. The architecture keeps deterministic classifiers and string matching where they work, sends low-confidence or inferential cases to LLMs, uses knowledge-graph retrieval to reduce duplicate entities and hallucination, and fine-tunes from generated annotations for scalable inference.

3 supporting quotes
“Building an in-house attribute extraction/tagging model from scratch requires a significant amount of labeled training data to reach the desired accuracy.”article, stated
“Because of their broad knowledge, LLMs can perform NLP with reasonable accuracy without requiring many, if any, labeled examples.”article, stated
“Choosing examples based on embedding similarity is advantageous over random selection because the selected examples are more likely to be relevant to the assigned task and reduces hallucination.”article, stated

Models used: GPT-4 · OpenAI embeddings · Google Bard · Meta Llama · multimodal LLMs

Scale

Alternatives considered (roads not taken)

Manual contract-operator SKU enrichment

Rejected because: Manual enrichment was slow, expensive, inaccurate enough to require a second audit, and hard to scale as the catalog expanded.

“outsourcing this task leads to long turnaround times, high costs, and so many inaccuracies that a second human must audit the results generated by the first.”article, stated

Build every extractor from scratch with labeled data

Rejected because: Cold-start labeling delays model development and catalog onboarding.

“Data collection slows model development, delays adding new items to the active catalog, and creates high operator costs.”article, stated

Organic string matching only

Rejected because: String matching had high precision but missed misspellings, omitted keywords, and variant presentations.

“This approach offered the highest precision and decent coverage, but it missed cases where ‚Äúorganic‚Äù is misspelled / dropped or has a slightly different presentation in the data.”article, stated

Random in-context examples for GPT-4 labeling

Rejected because: Embedding-similar examples are more relevant and reduce hallucination.

“Choosing examples based on embedding similarity is advantageous over random selection because the selected examples are more likely to be relevant to the assigned task and reduces hallucination.”article, stated

Patterns

rag-retrieval (retrieval)

DoorDash retrieved similar golden annotations to supply GPT-4 with relevant in-context examples for labeling unannotated SKUs.

“We used LLMs and retrieval augmented generation, or RAG, to accelerate label annotations.”article, stated

fine-tuning-adaptation (modeling)

Generated annotations from GPT-4 became training data for a model used for more scalable inference.

“Ultimately, the generated annotations are used to fine-tune an LLM for more scalable inference.”article, stated

tool-use-orchestration (orchestration)

The organic-labeling pipeline used an LLM agent to search external product information and hand results to another LLM for reasoning.

“LLMs conduct online searches of product information and pipe the search results to another LLM for reasoning.”article, stated

confidence-thresholding (routing)

Only SKUs the in-house classifier could not tag confidently were routed to the LLM extraction path.

“SKUs that cannot be tagged confidently to one of the existing brands are passed to an LLM for brand extraction”article, stated

data-quality-loop (data)

New brand annotations enter the knowledge graph and retrain the classifier, improving future catalog coverage.

“The new brand enters our knowledge graph and the in-house classifier is retrained with the new annotations”article, stated

Tradeoffs

Route low-confidence classifier cases to LLMs

Chose a classifier-first pipeline with LLM fallback over using LLM extraction for every SKU from the start.

Rationale: The existing classifier handles known brands, while LLMs address taxonomy gaps and cold-start cases.

“SKUs that cannot be tagged confidently to one of the existing brands are passed to an LLM for brand extraction”article, stated

Use string matching where it has high precision

Chose string matching plus LLM reasoning and agents for organic labels over a pure LLM-only labeling path.

Rationale: String matching was precise for obvious cases, while LLMs expanded coverage where strings failed.

“This approach offered the highest precision and decent coverage”article, stated

Fine-tune after generating annotations

Chose GPT-4 and RAG-generated labels as training data for scalable inference over using GPT-4 directly for all future inference.

Rationale: The source states generated annotations are used to fine-tune for scalability after RAG-assisted labeling.

“Ultimately, the generated annotations are used to fine-tune an LLM for more scalable inference.”article, stated

Gotchas (what bit them)

Manual catalog enrichment did not scale.

Root cause: Operator-led tagging had high cost, long turnaround, and enough inaccuracies to need another human audit.

Resolution: Use LLM-powered extraction pipelines to auto-enrich SKU data while keeping classifier and graph feedback loops.

“Historically, this SKU enrichment of extracting and tagging attributes has been a purely manual process led by contract operators.”article, stated

Brand taxonomies always have coverage gaps.

Root cause: New brands continuously appear, so a fixed taxonomy is never complete.

Resolution: Proactively extract new brands, check duplicates with the knowledge graph, ingest accepted brands, and retrain.

“As the product spectrum expands, new brands must be ingested to close any coverage gaps.”article, stated

Text-only attributes miss information hidden by abbreviations and messy product descriptions.

Root cause: Written product descriptions contain abstraction and abbreviation, while images can be more consistent across merchants.

Resolution: Explore multimodal extraction with Visual QA, Chat + OCR, and in-store product photos.

“A challenge with this approach, however, is the presence of abstraction and abbreviations within written product descriptions.”article, stated

Evaluation

The article describes quality qualitatively through coverage, precision, hallucination reduction, and downstream catalog utility rather than giving a full benchmark table.

Metrics: better than human precision for LLM organic reasoning · coverage boosted by LLM agents · embedding-similar examples reduce hallucination versus random examples

“This approach improved coverage by addressing major challenges faced by string matching and has better than human precision.”article, stated
“This approach further boosted our coverage.”article, stated

Outcomes

The source emphasizes catalog expansion and infinite brand coverage needs but does not publish exact SKU-volume or production-lift numbers.

“Enhancing coverage of organic labels enabled us to launch item carousels that target customers' with strong organic affinity”article, stated
“Attributes such as brand and organic tag are important features in our personalized ranking models”article, stated