Rebuilding Query Understanding for E-Commerce Search with LLMs

instacart · E-commerce · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce search query understanding

Instacart needed to rebuild query understanding for millions of imperfect grocery searches because traditional QU models struggled with broad and long-tail queries, noisy labels, and a collection of task-specific systems that were difficult to scale and evolve.

Constraints:

• Queries are upstream of purchase actions, so clean direct labels are scarce.
• Long-tail and highly specific searches have sparse historical engagement data.
• The platform needed real-time inference for live search, not only offline batch processing.
• Search improvements had to manage latency, cost, recall, precision, and hallucination risk.
• Outputs had to align with Instacart's product taxonomy and catalog constraints.

Architecture

Instacart moved QU to a layered LLM strategy. It uses context-engineered RAG and guardrails for category classification and head-query semantic role labeling, specialized rewrite prompts for substitutes/broader queries/synonyms, and a teacher/student SRL system where an offline RAG pipeline creates validated tags, a cache, and training data for a fine-tuned Llama-3-8B real-time model serving tail queries.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Search query stream"] --> B["Category classification pipeline"]
  A --> C["Query rewrite pipeline"]
  A --> D["SRL cache router"]
  B --> E["Converted category retriever"]
  B --> F["LLM re-ranker"]
  B --> G["Semantic similarity guardrail"]
  C --> G
  C --> H["Session engagement signals"]
  D --> I["Head-query tag cache"]
  D --> J["Fine-tuned real-time SRL model"]
  K["Offline RAG teacher"] --> L["Historical conversion data"]
  K --> M["Product catalog embeddings"]
  K --> N["Post-processing catalog guardrail"]
  K --> I
  K --> O["Training dataset"]
  O --> J
  J --> P["GPU serving infrastructure"]
  B --> Q["Search ranking and retrieval"]
  C --> Q
  D --> Q

Components

Data flow

Search queries first flow through LLM-powered QU tasks. For category classification, candidate categories are retrieved, an LLM re-ranks them with Instacart context, and semantic-similarity guardrails filter irrelevant outputs. For rewrites, dedicated prompts generate substitute, broader-query, and synonym rewrites, then guardrails validate relevance. For SRL, an offline RAG teacher enriches prompts with conversion and catalog context, validates tags, writes a low-latency cache for head queries, and produces training data for a fine-tuned real-time student model that handles cache misses and long-tail traffic.

Why this architecture

Instacart chose a layered architecture because different QU tasks had different latency and knowledge requirements. RAG/context engineering gives the model business context without retraining, guardrails protect taxonomy relevance, offline teacher processing can spend more compute on head-query quality, and fine-tuning distills that quality into a fast real-time model for the long tail.

Models used: Large Language Models · Llama-3-8B · FastText · LoRA fine-tuned model

Scale

• Query understanding helps millions of customers find items on Instacart.
  “Query Understanding (QU), is the intent engine that helps millions of customers find what they need on Instacart.”article, stated
• Instacart's taxonomy structures billions of items.
  “Instacart’s catalog is organized into a vast, hierarchical product taxonomy that structures billions of items”article, stated
• Legacy query rewrites covered only 50% of search traffic.
  “Our legacy system mined candidate rewrites from user session data, but this approach was limited, covering only 50% of search traffic”article, stated
• Structured LLM rewrites increased coverage to over 95% with 90%+ precision.
  “This structured approach increased our query rewrite coverage to over 95% with 90%+ precision across all three types.”article, stated
• Out-of-the-box real-time latency was nearly 700ms on A100 GPUs and was reduced to a 300ms target.
  “The out-of-the-box latency was nearly 700ms with A100 GPU. We reduced latency through a series of crucial optimizations:”article, stated

  “Merging the LoRA adapter weights directly into the base model and upgrading to H100 GPUs got us to our 300ms target.”article, stated
• The real-time LLM improved bottom-2% query quality and reduced average scroll depth by 6%.
  “the real-time LLM meaningfully improved search quality for the bottom 2% of queries. With the new SRL tagging for the tail queries, we reduce “average scroll depth” by 6%”article, stated
• The system serves millions of cold-start queries weekly and reduced poor-search-result complaints by 50%.
  “The system is now live, serving millions of cold-start queries weekly and reducing user complaints related to poor search results for tail queries by 50%.”article, stated

Alternatives considered (roads not taken)

Traditional bespoke QU models for each task

Rejected because: They struggled with long-tail generalization, required noisy labels, and created multiple systems with separate pipelines and serving architecture.

A simple single-model rewrite prompt

Rejected because: It was too ambiguous and generated valid synonyms that were not useful product-discovery rewrites.

Running the large offline LLM for every query at request time

Rejected because: Live long-tail queries needed fast and cost-effective inference, so Instacart cached head-query tags and trained a smaller real-time student model for cache misses.

Deploying the FP8 quantized model

Rejected because: Although it reduced latency, it slightly reduced recall, so Instacart deployed the unquantized model to prioritize quality.

Patterns

context-engineering (retrieval)

Instacart injected conversion history, catalog data, product similarity, and engagement signals into prompts so LLMs could behave like grocery-domain specialists.

rag-retrieval (retrieval)

The offline teacher uses retrieval-augmented generation to enrich prompts for high-frequency queries before generating tags and training data.

guardrails-io-filtering (safety-quality)

Post-processing guardrails validate model outputs against semantic relevance, product taxonomy, and catalog constraints before they affect search.

fine-tuning-adaptation (model-adaptation)

Instacart fine-tuned a smaller Llama-3-8B model with LoRA on offline teacher-generated data so it could serve long-tail SRL queries in real time.

model-distillation (model-adaptation)

The offline teacher pipeline produced a curriculum that distilled larger-model and domain-context behavior into the faster student model.

semantic-caching (serving)

Head queries are served from a validated low-latency cache, while cache misses route to the fine-tuned real-time model.

latency-optimization (serving)

The serving path reduced model latency with adapter merging, H100 GPUs, and selective avoidance of quality-hurting quantization.

Tradeoffs

Domain adaptation strategy

Chose fine-tuning plus context engineering and guardrails over prompting alone or another bespoke traditional model.

Rationale: The team found a hierarchy where fine-tuning most deeply embeds domain expertise, with RAG and prompting as progressively lighter adaptation layers.

Head-query and tail-query serving

Chose offline teacher plus cache for head queries and fine-tuned student for tail queries over one expensive live LLM path for all queries.

Rationale: This split gave high-quality processing where latency did not matter and fast inference where live search demanded it.

Rewrite generation design

Chose three specialized rewrite prompts with guardrails over one broad recall-enhancement prompt.

Rationale: The broad prompt was ambiguous; specialized prompts made rewrite intent explicit and improved coverage/precision.

Quantization deployment

Chose unquantized model over FP8 quantized model.

Rationale: The latency win was not worth the recall loss for production search quality.

Gotchas (what bit them)

Traditional labels were noisy because QU sits upstream of conversion.

Root cause: User behavior can imply misleading pseudo-labels, such as a user searching for one item and buying another.

Resolution: Instacart used LLMs with domain context and guardrails rather than relying only on engagement-trained traditional models.

Long-tail queries broke models trained mostly on engagement data.

Root cause: Rare, specific queries have limited historical clicks or conversions, so conventional models generalize poorly.

Resolution: The hybrid SRL system uses an offline teacher to create data and a fine-tuned real-time student for tail cache misses.

A simple rewrite prompt produced technically valid but commercially unhelpful rewrites.

Root cause: Recall-enhancement was under-specified, so the model returned synonyms instead of useful discovery alternatives.

Resolution: Instacart split rewrites into substitute, broader-query, and synonym prompts with COT, few-shot examples, and relevance filters.

Real-time LLM latency was initially too high for production search.

Root cause: The fine-tuned model still had nearly 700ms latency on A100 GPUs out of the box.

Resolution: Instacart merged adapters into the base model, upgraded to H100 GPUs, avoided recall-damaging quantization, and autoscaled GPU capacity.

Quantization improved speed but hurt quality.

Root cause: FP8 quantization cut latency at the cost of a slight recall drop.

Resolution: The production deployment kept the unquantized model to prioritize search quality.

Evaluation

Instacart evaluated the QU rebuild with task-level precision/recall/F1, rewrite coverage/precision, serving latency, A/B search-quality metrics, scroll depth, cold-start serving volume, and user complaint reduction.

Metrics: Rewrite coverage over 95% with 90%+ precision · Fine-tuned 8B model precision 96.4% versus 95.4% baseline · Fine-tuned 8B model recall 95% versus 96.2% baseline · Fine-tuned 8B model F1 95.7% versus 95.8% baseline · Latency reduced from nearly 700ms on A100 toward 300ms on H100 with adapter merging · Average scroll depth reduced by 6% · Poor-tail-search-result complaints reduced by 50%

Outcomes

• Query rewrite coverage improved from 50% of traffic to over 95% with 90%+ precision.
• The fine-tuned Llama-3-8B model reached near-frontier F1 with higher precision than baseline.
• Adapter merging and H100 serving brought real-time latency to the 300ms target.
• A/B testing showed improved bottom-2% query quality and 6% lower average scroll depth.
• The live system serves millions of cold-start queries weekly and reduced poor-search-result complaints for tail queries by 50%.
• Instacart established a reusable foundation for fine-tuning smaller open-source models for future search and discovery experiences.

The system operates in a high-scale grocery search environment with millions of customers, billions of catalog items, millions of weekly cold-start queries, and a hard real-time latency budget.