Enhancing E-commerce Search with LLMs at Scale

instacart · E-commerce · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Grocery e-commerce search

Instacart needed to improve grocery search for tail queries, product attributes, query intent, and merchandising across a huge catalog while managing LLM cost, latency, and evaluation risk.

Constraints:

Tail queries have little behavioral data and traditional models perform poorly on them.
Instacart has millions of tail queries and a massive multi-category grocery catalog.
Product attributes span thousands of categories and hundreds of attributes per category.
LLM calls can be costly and too slow for online search if used indiscriminately.
LLMs can misinterpret queries when they lack domain knowledge.

Architecture

Instacart uses LLMs selectively within the search stack. Offline LLM processing extracts product concepts and attributes from product descriptions and images, feeds those signals into query understanding, ranking, and merchandising, and avoids live latency for tasks that can be precomputed. For vague or complex online queries, the system can consider LLM boosts, but production design emphasizes deciding which queries are eligible and whether work belongs online or offline.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user-search-query["User search query"] --> query-understanding-layer["Query understanding layer"]
  product-catalog-data["Product catalog data"] --> offline-llm-attribute-extraction["Offline LLM attribute extraction"]
  offline-llm-attribute-extraction --> query-understanding-layer
  offline-llm-attribute-extraction --> search-ranking-layer["Search ranking layer"]
  offline-llm-attribute-extraction --> merchandising-workflows["Merchandising workflows"]
  online-llm-eligibility-gate["Online LLM eligibility gate"] --> query-understanding-layer
  query-understanding-layer --> search-ranking-layer
  search-ranking-layer --> search-results-page["Search results page"]
  merchandising-workflows --> search-results-page
  search-results-page --> evaluation-loop["Evaluation loop"]
  evaluation-loop --> offline-llm-attribute-extraction

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`User search query`	Customer grocery search input, including tail, vague, and complex intents.	`Query understanding layer`
`Query understanding layer`	Uses rewrites, intent, and attribute signals to improve recall and ranking.	`Search ranking layer`
`Product catalog data`	Product descriptions, images, retailer data, and third-party information.	`Offline LLM attribute extraction`
`Offline LLM attribute extraction`	Extracts product concepts and attributes outside the live request path.	`Query understanding layer`, `Search ranking layer`, `Merchandising workflows`
`Online LLM eligibility gate`	Decides whether a live query deserves an LLM call based on complexity and value.	`Query understanding layer`
`Search ranking layer`	Ranks and recalls products using query and product-attribute signals.	`Search results page`
`Merchandising workflows`	Uses LLM product understanding to create or improve collections and sponsored discovery.	`Search results page`
`Search results page`	Displays relevant products and discovery content to customers.	`Evaluation loop`
`Evaluation loop`	Measures conversion, discovery, relevance, and human or LLM-based judgments.	`Offline LLM attribute extraction`

Data flow

Product information, descriptions, images, and query examples are processed by LLMs to extract product attributes, concepts, and query tags. Offline outputs feed query understanding, ranking, and merchandising; online paths are reserved for cases where live LLM understanding is necessary and worth the latency/cost tradeoff.

“once you uh face this back into like uh layers like query understanding and ranking the entire search becomes very powerful”transcript, stated

“this can be done in an offline Manner and once this is extracted in offline manner they can be integrated back into our career understanding our ranking”transcript, stated

Why this architecture

Instacart treats LLMs as targeted search enhancers rather than a blanket replacement for traditional search. LLMs help where traditional systems struggle, such as tail queries and attribute extraction, but offline processing and eligibility gates protect production cost and latency.

“it's very important for you to understand when to use an llm call versus like when to use your traditional search”transcript, stated

“first thing is the cost and latency obviously it's going to be costly if you were to call this llm for everything”transcript, stated

Models used: LLMs · GPT-V · traditional NLP models

Scale

Instacart had around 20 million tail queries.
“we have around 20 million tail queries today at instacart and that keeps on growing from time to time”transcript, stated
Instacart operates with 400 national, regional, and local retailer brands.
“instacut operates at like uh operates 400 National regional and local retailer Brands out of 8,000 stores”transcript, stated
Instacart operates across 8,000 stores and 14,000 US cities.
“400 National regional and local retailer Brands out of 8,000 stores across 14,000 cities in us”transcript, stated
The catalog spans over a billion products.
“we operate over a billion on over a billion of products”transcript, stated
The catalog spans several thousand categories with hundreds of attributes per category.
“we deal with like several hundred several thousand categories and each of these categories can have hundreds of attributes”transcript, stated

Alternatives considered (roads not taken)

Build separate NLP models for every search task.

Rejected because: Separate spell correction, entity linking, and intent models require task-specific design and data, making development slow and costly.

“every task needs like NLP model for example spell correction or like entity linking or uh int classification right everything needs a a model on its own”transcript, stated

Manually extract product attributes with domain experts.

Rejected because: Manual extraction across thousands of categories and attributes is costly, unscalable, and error-prone.

“domain experts who manually go through this and like extract them but this is as you can imagine this is not at all scalable or this and it's very costly and also error prone”transcript, stated

Call an LLM for every query online.

Rejected because: Cost and latency are high, and simple queries like milk may not need LLM help.

“you might not need a llm boost for a query like milk because we already have a pretty good understanding of it”transcript, stated

Patterns

`batch-offline-processing` (operations)

Product attribute extraction is done offline and then integrated into query understanding and ranking.

“product extraction we're talking about right this can be done in an offline Manner”transcript, stated

`cost-aware-routing` (serving)

The search system decides when LLM calls are worth the latency and cost versus traditional search.

“it's very important for you to understand when to use an llm call versus like when to use your traditional search”transcript, stated

`query-rewriting` (retrieval)

LLMs support query understanding, tagging, and interpreting complex shopping intents.

“query understanding has models like query rewrites which essentially says that hey the user search for qts”transcript, stated

`human-in-the-loop-gate` (evaluation)

Human raters judge whether products are relevant for a query, though the speaker notes LLMs may help automate annotation.

“we go through something called human evaluation or like with the human rat really evaluating whether a given product is like uh relevant to a given query”transcript, stated

Tradeoffs

LLM placement

Chose Offline processing where possible and selective online use. over LLM calls for every live query..

Rationale: Offline processing protects latency and cost, while online calls are reserved for complex cases.

“is it really essential that I need to make an llm call in real time or can some of this be processed offline”transcript, stated

Search architecture

Chose LLMs augment query understanding, ranking, merchandising, and extraction. over Replacing the whole search stack with LLM generation..

Rationale: LLM-derived signals are fed back into existing search layers where they improve recall and ranking.

“once this is extracted in offline manner they can be integrated back into our career understanding our ranking in an offline fashion”transcript, stated

Gotchas (what bit them)

Tail queries lack enough behavioral data for traditional models.

Root cause: Rare queries do not appear often, so existing models perform poorly on them.

Resolution: Use LLM understanding for tail-query and complex-intent cases.

“tail queries are essentially queries that we don't see often they don't perform very well because there's not a lot of data about it”transcript, stated

Manual attribute extraction cannot scale with catalog breadth.

Root cause: Categories and attributes are numerous, and manual domain-expert extraction is expensive and error-prone.

Resolution: Use LLMs to extract product concepts and attributes from descriptions and images.

“llms are actually a GameChanger this is because the task of like giving llms a bunch of product information product description and asking you to extract product Concepts”transcript, stated

LLMs can misread grocery intent.

Root cause: Missing domain knowledge can make the model return recipe-style answers when the user wants purchasable products.

Resolution: Evaluate with online product-discovery and conversion metrics plus relevance judgments.

“llm might be giving recipes to generate that instead of like where whereas the user is looking to buy something like a product that's readily available for purchase”transcript, stated

Evaluation

Instacart evaluates LLM search improvements through online evaluation, product discovery, search conversion, human relevance ratings, and potentially LLM-assisted annotation.

Metrics: Product discovery · Search conversion rate · Human relevance ratings · Latency · Cost

“if the new product Discovery increases it shows that hey llms are doing a great job”transcript, stated

“increase in search conversion rate shows that hey users are really interacting I mean well with the llms”transcript, stated

Outcomes

LLMs are positioned to improve tail-query handling, product attribute extraction, query tagging, and merchandising.
Offline extraction avoids latency hits while feeding query understanding and ranking.
Selective online LLM calls are reserved for vague and complex intents.
Evaluation focuses on discovery, conversion, relevance, cost, and latency.

Instacart cites around 20 million tail queries, 400 retailer brands, 8,000 stores, 14,000 US cities, over a billion products, and several thousand categories.

“there are a lot of new models coming into like there's a very big hope for all this uh human annotation task to be like uh moved on to the llms”transcript, stated