LLM-Enhanced Search and Discovery for Grocery E-commerce

instacart · E-commerce · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Grocery e-commerce search and discovery

Instacart needed better search query understanding and discovery for tail queries where traditional models lacked engagement data and general LLM knowledge did not match grocery-specific user intent.

Constraints:

• The query-to-category task maps a query to a multilabel taxonomy of about 10,000 labels, with about 6,000 commonly used.
• Traditional FastText, NPMI, and BERT-style models worked better for head and torso queries than long-tail queries.
• Naive LLM outputs reflected world knowledge rather than Instacart user behavior, such as interpreting protein as foods rather than supplements.
• Real-time LLM calls were constrained by latency and cost, especially for high-volume query traffic.

Architecture

Instacart used LLMs as targeted enhancements to query understanding and discovery rather than replacing the search stack. For query-to-category classification, the LLM receives the query plus top converting categories and other Instacart context, turning open-ended prediction into context-informed ranking. Head and torso query outputs are precomputed offline and cached, long-tail cases fall back to existing models or distilled Llama exploration, and generated discovery content is served from a feature store after batch LLM generation, ranking, and evaluation.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Query_understanding_module["Query understanding module"] --> Query_to_category_classifier["Query-to-category classifier"]
  Query_to_category_classifier --> Taxonomy["Taxonomy"]
  Taxonomy --> LLM_context_ranker["LLM context ranker"]
  Top_converting_category_context["Top converting category context"] --> LLM_context_ranker
  LLM_context_ranker --> Batch_precompute_cache["Batch precompute cache"]
  Batch_precompute_cache --> Feature_store["Feature store"]
  LLM_context_ranker --> Discovery_content_generator["Discovery content generator"]
  Discovery_content_generator --> Feature_store
  Feature_store --> Search_ranking_stack["Search ranking stack"]
  Query_understanding_module --> Search_ranking_stack
  LLM_context_ranker --> Evaluation_pipeline["Evaluation pipeline"]

Components

Data flow

Queries enter the query understanding module, which includes normalization, tagging, classification, and category classification. For category prediction, Instacart provides taxonomy and top converting categories as LLM context, ranks candidates, validates downstream, and precomputes outputs for frequent queries. For discovery content, historical search logs are batch-processed by LLMs with query metadata and behavior context, stored with candidate products, and served online through fast feature-store lookup into the existing retrieval/ranking stack.

Why this architecture

The architecture keeps low-latency search serving practical by batching and caching frequent LLM outputs, while injecting Instacart-specific behavior data to correct the mismatch between general world knowledge and grocery search intent.

Models used: LLMs · FastText-based neural network · NPMI co-occurrence model · BERT-based models · distilled Llama model · LLM-as-judge · PCTR/PCBR ranking models

Scale

• Instacart's taxonomy has about 10,000 labels, of which about 6,000 are commonly used.
  “our taxonomy has about 10,000 labels of it 6,000 are more commonly used.”transcript, stated
• Tail-query precision improved by 18 percentage points.
  “Our precision improved by our 18 percentage points and our recall improved by our 70 percentage points”transcript, stated
• Tail-query recall improved by 70 percentage points.
  “Our precision improved by our 18 percentage points and our recall improved by our 70 percentage points”transcript, stated
• The serving design precomputes head and torso query outputs offline and serves them with low latency from cache.
  “for all of the head and torso queries offline in a batch mode we were able to sort of uh cache all of this data”transcript, stated

Alternatives considered (roads not taken)

Use traditional FastText/NPMI/BERT models alone

Rejected because: They worked for head and torso queries but had low tail-query coverage and BERT latency did not justify the gains.

Ask the LLM for categories without Instacart behavior context

Rejected because: Manual outputs looked sensible but online tests showed that general world knowledge misread grocery intent.

Call the LLM online for every query

Rejected because: Latency and cost made real-time LLM calls impractical for serving all generated discovery content.

Patterns

semantic-caching (serving)

Instacart precomputes LLM outputs for head and torso queries offline and serves them from cache or a feature store to avoid online latency.

eval-driven-development (evaluation)

Instacart used online A/B tests, downstream retrieval/ranking results, business metrics, and LLM-as-judge evaluation to iterate on query and content quality.

llm-as-judge (evaluation)

The team used LLM-as-judge checks to evaluate generated content and query predictions against Instacart product requirements.

Tradeoffs

Use LLMs with domain context for query understanding.

Chose LLM predictions constrained by Instacart behavior data over Open-ended LLM category generation.

Rationale: Domain context aligned outputs with actual shopping intent and improved tail-query precision and recall.

Precompute frequent-query LLM outputs offline.

Chose Batch mode plus cache/feature-store serving over Online LLM calls for all queries.

Rationale: This preserved low-latency serving while still improving long-tail coverage and discovery content.

Use a hybrid model portfolio instead of consolidating immediately.

Chose Existing models, cached LLM outputs, and distilled Llama exploration over One unified LLM/SLM for all query understanding immediately.

Rationale: The source says model consolidation may improve consistency, but the production design still uses hybrid fallback while replacements are explored.

Gotchas (what bit them)

General LLM world knowledge conflicted with grocery-user intent.

Root cause: The LLM interpreted queries semantically in the general world rather than through Instacart shopping behavior.

Resolution: Inject top converting categories, query annotations, and subsequent-user-query metadata into prompts.

Tail queries lacked enough engagement data for existing models.

Root cause: Traditional models depended on engagement/co-occurrence signals that were sparse in the long tail.

Resolution: Use LLM world knowledge combined with Instacart context and explore distilled Llama models for long-tail replacement.

LLM-generated content that looked good did not necessarily drive engagement.

Root cause: Common-sense generated lists were not what users were actually looking for.

Resolution: Rework prompts with Instacart-specific metadata, ranking, business metrics, and evaluation checks.

Evaluation

Instacart evaluated query/category models through online A/B testing, downstream retrieval and ranking impact, tail-query precision and recall, engagement and revenue for discovery content, and LLM-as-judge checks for hallucination and product fit.

Metrics: online A/B test results · 18 percentage point precision improvement · 70 percentage point recall improvement · engagement · revenue · ranking recall · LLM-as-judge content evaluation

Outcomes

• Tail-query category precision improved by 18 percentage points.
• Tail-query category recall improved by 70 percentage points.
• Adding Instacart context to generated discovery content improved engagement and revenue.
• The team identified automated evaluation as more difficult and more important than expected.

Instacart's query-category taxonomy has about 10,000 labels, with about 6,000 commonly used, and the serving strategy separates head/torso batch caching from long-tail fallback.