LLMs for Enhanced Search Retrieval and Query Understanding

doordash · E-commerce · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce search retrieval and query understanding

DoorDash needed search retrieval to handle compound queries such as vegan chicken sandwich, where embedding similarity alone could retrieve semantically nearby but user-invalid results.

Constraints:

Architecture

DoorDash built an LLM-assisted query-understanding layer around its product and food knowledge graphs. Documents are annotated with taxonomy metadata and indexed; queries are segmented into structured taxonomy categories, linked to knowledge-graph concepts through ANN-retrieved candidate labels and LLM selection, then converted into retrieval logic such as MUST dietary filters and SHOULD flavor preferences. Rankers consume the new signals and online tests measure relevance, carousel trigger rate, conversion, and retrained-ranker gains.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  KG["Knowledge graph taxonomies"] --> Doc_annotation["Document metadata annotation"]
  Doc_annotation --> Search_index["Search index"]
  User_query["User query"] --> Segmenter["LLM query segmenter"]
  User_query --> ANN["ANN candidate retriever"]
  KG --> ANN
  Segmenter --> Linker["LLM entity linker"]
  ANN --> Linker
  Segmenter --> Validation["Post-processing validation"]
  Linker --> Validation
  Linker --> Retrieval_logic["Retrieval logic"]
  Search_index --> Retrieval_logic
  Retrieval_logic --> Rankers["Rankers"]
  Search_index --> Rankers
  Rankers --> Surfaces["Search result surfaces"]
  Validation --> Audit["Manual audit process"]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Knowledge graph taxonomies Controlled food and retail ontology for cuisines, dish types, dietary preferences, brands, categories, and related attributes. Document metadata annotation, ANN candidate retriever, LLM entity linker
Document metadata annotation Adds rich metadata tags to stores and items before indexing. Knowledge graph taxonomies, Search index
Search index Retrieval substrate containing indexed items, stores, and knowledge-graph-derived attributes. Document metadata annotation, Retrieval logic, Rankers
User query Search text with compound requirements such as dish type, dietary preference, quantity, or flavor. LLM query segmenter, ANN candidate retriever
LLM query segmenter Segments queries into meaningful pieces and maps them to structured taxonomy categories. User query, LLM entity linker, Post-processing validation
ANN candidate retriever Retrieves the closest taxonomy concepts for each query so the LLM chooses from valid candidates. User query, Knowledge graph taxonomies, LLM entity linker
LLM entity linker Selects the most appropriate knowledge-graph concept for each query segment from retrieved candidates. LLM query segmenter, ANN candidate retriever, Retrieval logic, Post-processing validation
Post-processing validation Checks segmentation and linked entities to prevent invalid or hallucinated final outputs. LLM query segmenter, LLM entity linker, Manual audit process
Retrieval logic Applies linked concepts as strict or flexible retrieval conditions such as dietary MUST and flavor SHOULD. LLM entity linker, Search index, Rankers
Rankers Order retrieved documents using the new query-understanding signals and updated engagement data. Search index, Retrieval logic, Search result surfaces
Search result surfaces User-facing result pages and popular dish carousels that expose the improved retrieval. Rankers
Manual audit process Annotator review of statistically significant samples for segmentation and entity-linking quality. Post-processing validation

Data flow

Catalog items and stores are annotated with knowledge-graph metadata before entering the search index. User queries pass through LLM segmentation, taxonomy classification, ANN candidate retrieval, and LLM entity linking. The linked concepts become retrieval constraints and ranking features, then downstream rankers order items or stores for surfaces such as popular dish carousels.

3 supporting quotes
“At DoorDash, documents refer to items or stores/restaurants, while queries are the search terms users enter into the search bar.”article, stated
“The query understanding module typically includes steps such as parsing and segmenting the query, annotating it with helpful information, linking it to specific concepts, and/or correcting spelling errors”article, stated
“This process ultimately generates a set of linked taxonomy concepts for each query that we can use directly to retrieve items from the search index.”article, stated

Why this architecture

DoorDash chose a constrained hybrid architecture because pure embedding similarity cannot enforce hard query intent, while unrestricted LLM outputs can hallucinate labels. Knowledge graph taxonomies give the LLM a controlled output space, ANN retrieval keeps prompts small and relevant, and traditional retrieval/ranking methods keep the system adaptable to novel queries.

3 supporting quotes
“For these keywords only the last set on that list matches the user intent exactly. But preferences may vary for different attributes.”article, stated
“We needed to develop a controlled vocabulary to create meaningful segmentations that are both factual and valuable for our retrieval system.”article, stated
“A hybrid approach strikes the right balance between memorization and generalization.”article, stated

Models used: LLMs · closed-source embedding models · pre-trained embedding models · in-house learned embedding models · BM25

Scale

Alternatives considered (roads not taken)

Embedding-only document similarity retrieval

Rejected because: Similarity retrieval could return vegan sandwiches, vegetarian sandwiches, or chicken sandwiches when the user specifically asked for vegan chicken sandwiches.

“a query such as “vegan chicken sandwich,” for which a retrieval system that relies on document similarity — such as an embedding-based system — could retrieve documents”article, stated
“For these keywords only the last set on that list matches the user intent exactly.”article, stated

Traditional PMI or n-gram query segmentation

Rejected because: Statistical segmentation struggles with ambiguous, overlapping, or context-dependent query relationships.

“Traditionally, query segmentation relies on methods such as pointwise mutual information (PMI) or n-gram analysis”article, stated
“These methods can be effective if the queries are relatively simple. They begin to fall short when dealing with complex queries that include multiple overlapping entities”article, stated

Let the LLM freely generate entity labels

Rejected because: Free generation can hallucinate concepts or mislabel segments; restricting choices to retrieved taxonomy candidates preserves controlled vocabulary consistency.

“this could mean mapping a query segment to a concept that doesn't exist in our knowledge graph or mislabeling it entirely.”article, stated
“This approach ensures that the model selects from concepts that already are part of our knowledge graph, maintaining consistency and accuracy in the mapping.”article, stated

Batch-precompute every possible query understanding result

Rejected because: DoorDash's dynamic query environment makes fixed-query memorization hard to scale and prone to maintenance and staleness issues.

“Using LLMs for batch inference on a fixed set of queries can provide highly accurate results.”article, stated
“As new queries emerge, especially in DoorDash’s dynamic environment, it becomes impractical to pre-process every possible query in a timely manner.”article, stated

Patterns

structured-output-contract (integration)

DoorDash prompted the LLM to output query segments mapped to specific taxonomy categories instead of arbitrary strings.

“we prompt it to provide a structured output mapping each meaningful word segment to one of our taxonomy categories”article, stated
“Our evaluations have shown that this approach results in more accurate segmentations, likely because the structured categories provide the model with additional context”article, stated

rag-retrieval (retrieval)

Entity linking uses retrieved taxonomy candidates as prompt context so the LLM selects from valid knowledge-graph concepts.

“To do this, we leverage retrieval-augmented generation, or RAG.”article, stated
“Then, using an ANN retrieval system, we retrieve the closest 100 taxonomy concepts, or candidate labels, for each search query.”article, stated

hybrid-search (retrieval)

The production search design combines keyword retrieval, LLM query understanding, knowledge-graph constraints, embeddings, BM25, and rule-based methods.

“a keyword-based retrieval system, combined with robust document and keyword understanding, can effectively enforce such rules as ensuring that only vegan items are retrieved.”article, stated
“By combining the approach we outline here with other methods that generalize well to new query-document pairs – including lightweight heuristics, statistical methods such as BM25, or more complex approaches like embedding retrieval”article, stated

safety-policy-enforcement (quality)

Dietary restrictions are modeled as strict MUST retrieval conditions so search does not violate important user constraints.

“This makes it easier to control what to retrieve by implementing a specific retrieval logic, such as making all dietary restrictions a MUST condition”article, stated

human-in-the-loop-gate (evaluation)

Annotators manually audit statistically significant samples to verify segmentation and entity-linking quality.

“After these post-processing steps, we perform manual audits on each batch of processed queries to measure the quality of our system.”article, stated
“Annotators review a statistically significant sample of the output to verify that query segments are correctly identified and accurately linked”article, stated

Tradeoffs

Use structured taxonomy outputs for segmentation

Chose category-aware LLM segmentation over arbitrary text chunks or PMI/ngram segments.

Rationale: Taxonomy categories gave the model context about relationships between segments and improved segmentation accuracy for complex queries.

"Instead of asking the model simply to find meaningful word segments such as:

[“small”, “no-milk”, “vanilla ice cream”] we prompt it to provide a structured output"

“Our evaluations have shown that this approach results in more accurate segmentations, likely because the structured categories provide the model with additional context”article, stated

Constrain entity linking with ANN candidates

Chose RAG over 100 nearest taxonomy concepts over free-form LLM concept generation.

Rationale: Candidate restriction reduces hallucinated or nonexistent knowledge-graph mappings while keeping the prompt within context limits.

“we retrieve the closest 100 taxonomy concepts, or candidate labels, for each search query. We need to do this because of context window limitations and to reduce the noise in the prompt”article, stated
“This approach ensures that the model selects from concepts that already are part of our knowledge graph, maintaining consistency and accuracy in the mapping.”article, stated

Blend memorized LLM batch outputs with generalizing retrieval methods

Chose hybrid LLM, embedding, statistical, and rule-based search over pre-processing every possible query with LLM batch inference.

Rationale: The LLM path is accurate on known query sets, but unseen queries require adaptable real-time methods.

“Using LLMs for batch inference on a fixed set of queries can provide highly accurate results.”article, stated
“A hybrid approach strikes the right balance between memorization and generalization.”article, stated

Gotchas (what bit them)

Embedding similarity can violate hard user constraints.

Root cause: Nearest-neighbor document similarity does not know that vegan is a mandatory dietary restriction while chicken may be a flexible protein attribute.

Resolution: Link query segments to knowledge-graph concepts and enforce dietary restrictions as MUST retrieval conditions.

“a consumer might be open to considering any vegan sandwich as an alternative but would reject a chicken sandwich that is not vegan; dietary restrictions often take precedence”article, stated
“making all dietary restrictions a MUST condition and allowing flexibility of less strict attributes such as flavors as a SHOULD condition.”article, stated

LLM segmentation and entity linking can hallucinate invalid search concepts.

Root cause: Unconstrained LLMs may invent taxonomy labels or map segments to concepts absent from the knowledge graph.

Resolution: Constrain outputs with structured taxonomy categories, ANN-retrieved candidates, post-processing validation, and manual audits.

“One problem with LLMs, however, is that they are prone to hallucinations.”article, stated
“we employ techniques that constrain the model's output to include only concepts within our controlled vocabulary – in other words, our taxonomy concepts.”article, stated

Batch inference on fixed queries can become stale and unscalable.

Root cause: Food-delivery queries, language, and knowledge-graph concepts change continuously.

Resolution: Combine accurate LLM batch processing with methods that generalize to unseen queries in real time.

"There are serious drawbacks to relying solely on memorization, including:

“Fortunately, other methods generalize well to unseen queries, such as embedding retrieval, traditional statistical models, and other rule-based systems”article, stated

Search rankers must adapt after query understanding changes retrieval behavior.

Root cause: Improved retrieval introduces new signals and new engagement patterns, so the ranking layer cannot remain static.

Resolution: Expose query-understanding signals to rankers and retrain with more diverse engagement generated by the improved retrieval system.

“After introducing the new query understanding signals, we needed to make them available to the rankers.”article, stated
“with new and more diverse engagement coming in from the improved retrieval systems, we could retrain our ranker with a more comprehensive dataset.”article, stated

Evaluation

DoorDash used model evaluation for segmentation accuracy, post-processing validation, manual audits of statistically significant query samples, and online business/relevance tests after rankers consumed the new signals.

Metrics: segmentation hallucination rate under 1% · popular dish carousel trigger rate · whole page relevance for dish-intent queries · same-day conversions · ranker WPR improvement after retraining

3 supporting quotes
“Even though the hallucination rate on the segmentation process is low — less than one percent — we also benefit from the immediate classification of the output”article, stated
“After these post-processing steps, we perform manual audits on each batch of processed queries to measure the quality of our system.”article, stated
“relevance and business metrics rose, as reflected in our online tests (see additional details below).”article, stated

Outcomes

The system operates inside DoorDash production search across dynamic food and retail catalogs; scale is reflected through online search relevance, carousel triggering, conversion, and ranker retraining metrics rather than a one-off offline demo.

4 supporting quotes
“we observed nearly a 30% increase over our baseline, which also means we are aligning search results more closely with consumer intent”article, stated
“Our approach led to a more than two percent increase in WPR for dish-intent queries, indicating that users were seeing more relevant dishes in general.”article, stated
“Online testing also showed that increased relevance aligns with an increase in engagement and conversion. We observed a rise in same-day conversions”article, stated
“The new ranker version further improved relevance — as demonstrated by a 1.6% increase in WPR”article, stated