Revamping Query Understanding with LLMs in E-commerce Search

instacart · E-commerce · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce search query understanding

Instacart needed to improve query understanding for broad, tail, and creatively phrased grocery searches because traditional task-specific ML systems struggled with noisy labels, sparse engagement data, and operational complexity.

Constraints:

• Query understanding happens upstream of clicks and conversions, so labels are indirect and noisy.
• Tail queries are effectively infinite and cannot all be processed offline by large models.
• Search serving has low-hundreds-of-milliseconds latency requirements.
• Generated categories, rewrites, and tags must stay aligned with Instacart's product taxonomy and catalog.
• The system had to manage quality, latency, cost, and GPU capacity at live-search scale.

Architecture

Instacart built a layered LLM query-understanding system. It uses context-engineered RAG to inject conversion history and catalog context, guardrails to filter hallucinated or taxonomy-incompatible outputs, specialized rewrite prompts, and an offline teacher pipeline that populates a head-query cache while generating training data for a fine-tuned Llama-3-8B real-time student model for long-tail cache misses.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Customer search queries"] --> B["Category classification pipeline"]
  A --> F["Query rewrite pipeline"]
  A --> G["SRL cache router"]
  B --> C["Converted category retriever"]
  B --> D["LLM reranker"]
  B --> E["Semantic similarity guardrail"]
  F --> E
  G --> I["Head-query tag cache"]
  G --> K["Fine-tuned real-time SRL model"]
  H["Offline RAG teacher"] --> D
  H --> I
  H --> J["Training dataset"]
  J --> K
  K --> L["GPU serving infrastructure"]
  K --> M["Search ranking and retrieval"]

Components

Data flow

Search queries enter QU tasks. Category classification retrieves top converted categories, reranks them with an LLM and injected Instacart context, then filters predictions through semantic-similarity guardrails. Query rewrite uses dedicated prompts for substitutes, broader queries, and synonyms. Semantic role labeling uses an offline RAG teacher for high-quality head-query tags, writes those tags into a cache, and trains a fine-tuned real-time model that handles cache misses.

Why this architecture

Instacart used a layered architecture because each part of QU had different economics. RAG and context engineering could inject domain knowledge for offline or high-value tasks, guardrails protected taxonomy correctness, caching made common queries cheap, and a fine-tuned smaller model preserved enough teacher quality for real-time long-tail serving.

Models used: Large Language Models · Llama-3-8B · FastText · LoRA fine-tuned model

Scale

• Query understanding supports millions of Instacart customers.
  “Query Understanding (QU), is the intent engine that helps millions of customers find what they need on Instacart.”article, stated
• Instacart's product taxonomy structures billions of catalog items.
  “Instacart’s catalog is organized into a vast, hierarchical product taxonomy that structures billions of items”article, stated
• Legacy rewrites covered only 50% of search traffic.
  “Our legacy system mined candidate rewrites from user session data, but this approach was limited, covering only 50% of search traffic”article, stated
• The production system serves millions of cold-start queries weekly.
  “The system is now live, serving millions of cold-start queries weekly and reducing user complaints related to poor search results for tail queries by 50%.”article, stated

Alternatives considered (roads not taken)

Continue building bespoke traditional QU models

Rejected because: The prior approach created separate pipelines and serving stacks for each task and struggled with long-tail language.

Run an expensive large model online for every semantic role label query

Rejected because: The long tail is effectively infinite and online LLM processing is too expensive for every query.

Use one generic rewrite prompt

Rejected because: A broad recall-enhancement prompt was too ambiguous, so Instacart split the job into substitute, broader-query, and synonym prompts.

Deploy FP8 quantization for extra latency reduction

Rejected because: The extra 10% latency improvement caused a slight recall drop, so production used the unquantized model.

Patterns

context-engineering (retrieval)

Instacart injected catalog, conversion, brand, and query-performance context into prompts so LLMs could act as grocery-domain specialists.

guardrails-io-filtering (safety-quality)

Post-processing guardrails filter hallucinations and enforce taxonomy, semantic relevance, and catalog validity before QU outputs reach search.

fine-tuning-adaptation (model-adaptation)

A Llama-3-8B student model was fine-tuned with LoRA on offline teacher data to serve long-tail SRL in real time.

semantic-caching (serving)

Validated tags for common head queries are served from a cache, while cache misses route to the real-time model.

model-distillation (model-adaptation)

The offline teacher's larger-model behavior and domain context were distilled into the smaller real-time student.

latency-optimization (serving)

Adapter merging, H100 GPUs, autoscaling, and avoiding recall-damaging quantization brought live inference to the latency target.

Tradeoffs

Domain adaptation ladder

Chose fine-tuning plus context engineering and guardrails over prompting alone.

Rationale: Instacart found fine-tuning most effective, context engineering useful, and prompting the lightest layer, with each carrying different engineering costs.

Head and tail serving split

Chose offline teacher/cache for head queries and real-time student for tail queries over one live large-model path for every query.

Rationale: The split spends compute where latency does not matter and uses the smaller model for real-time cache misses.

Latency versus recall

Chose unquantized model over FP8 quantized model.

Rationale: The additional latency reduction was not worth the recall loss for production search.

Gotchas (what bit them)

Traditional QU labels were noisy because search intent is upstream of purchase behavior.

Root cause: Pseudo-labels from behavior can be misleading when a user searches for one thing but buys another.

Resolution: Instacart used domain-context LLM inference and guardrails instead of relying only on engagement-trained labels.

A single broad rewrite prompt was under-specified.

Root cause: Synonyms, substitutes, and broader queries serve different user needs even though all improve recall.

Resolution: Instacart built three specialized prompts with chain-of-thought, few-shot examples, and relevance filters.

The real-time student model was too slow out of the box.

Root cause: Initial serving latency was nearly 700ms on A100 GPUs.

Resolution: The team merged LoRA adapters, upgraded to H100 GPUs, evaluated FP8, and autoscaled GPU capacity.

Quantization improved speed but reduced recall.

Root cause: FP8 saved latency at the cost of search-quality signal.

Resolution: Instacart deployed the unquantized model to prioritize quality.

Evaluation

Instacart evaluated the QU rebuild through rewrite coverage and precision, model precision/recall/F1, latency, A/B search-quality measurements, scroll depth, cold-start query volume, and complaint reduction.

Metrics: Query rewrite coverage over 95% with 90%+ precision · Fine-tuned 8B precision 96.4% versus 95.4% baseline · Fine-tuned 8B recall 95% versus 96.2% baseline · Fine-tuned 8B F1 95.7% versus 95.8% baseline · Latency reduced from nearly 700ms to a 300ms target · Average scroll depth reduced by 6% · Poor tail-query complaint volume reduced by 50%

Outcomes

• Rewrite coverage increased to over 95% with 90%+ precision.
• The fine-tuned 8B model matched larger-model F1 while improving precision.
• Adapter merging and H100 serving achieved the 300ms latency target.
• Tail-query average scroll depth fell by 6%.
• Poor-tail-search-result complaints fell by 50%.
• Instacart established a reusable capability for fine-tuning smaller open-source models for search.

The system operates over millions of customers, billions of catalog items, live low-latency search traffic, and millions of cold-start queries per week.