Using LLMs to Enhance Search Discovery and Recommendations

instacart · E-commerce · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Grocery search discovery and recommendations

Instacart needed to add inspirational discovery content to search results without degrading relevance, latency, page layout, or revenue outcomes.

Constraints:

Traditional related-items logic produced weak alternatives for narrow queries.
Discovery content had to suggest substitutes and complementary products, not just exact matches.
LLM generation needed Instacart-specific context such as query understanding and conversion data.
Serving had to avoid request-time LLM latency and cost.
More page content created ranking and clutter tradeoffs.

Architecture

Instacart precomputes LLM-generated discovery content offline. Batch jobs enrich historical queries with query-understanding annotations, conversion patterns, consecutive-search signals, and prompt templates; an LLM generates substitutes and complementary shopping lists; generated terms are mapped through Instacart's search engine into products; post-processing removes duplicates and diversifies results; a key-value store serves content at query time; and a Whole Page Ranker positions carousels on the page.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  search-logs-and-query-metadata["Search logs and query metadata"] --> offline-prompt-generation["Offline prompt generation"]
  offline-prompt-generation --> llm-content-generator["LLM content generator"]
  llm-content-generator --> llm-content-store["LLM content store"]
  llm-content-generator --> product-mapping-search["Product mapping search"]
  product-mapping-search --> post-processing-and-diversification["Post-processing and diversification"]
  llm-content-store --> runtime-search-service["Runtime search service"]
  post-processing-and-diversification --> runtime-search-service
  runtime-search-service --> whole-page-ranker["Whole Page Ranker"]
  llm-content-generator --> llm-judge-evaluator["LLM judge evaluator"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Search logs and query metadata`	Provide historical queries, engagement, conversion, and query-understanding signals.	`Offline prompt generation`
`Offline prompt generation`	Populates prompt templates with enriched query context.	`Search logs and query metadata`, `LLM content generator`
`LLM content generator`	Produces substitute, complementary, and themed shopping-list concepts.	`Offline prompt generation`, `LLM content store`
`LLM content store`	Stores generated content by query for low-latency serving.	`LLM content generator`, `Runtime search service`
`Product mapping search`	Maps generated item concepts to actual catalog products through existing search.	`LLM content generator`, `Post-processing and diversification`
`Post-processing and diversification`	Removes duplicates, filters irrelevant products, and reranks for variety.	`Product mapping search`, `Runtime search service`
`Runtime search service`	Serves standard results and LLM-generated inspirational carousels for user queries.	`LLM content store`, `Post-processing and diversification`, `Whole Page Ranker`
`Whole Page Ranker`	Positions discovery content on the page while balancing relevance and revenue.	`Runtime search service`
`LLM judge evaluator`	Scores generated complementary/substitute content quality at scale.	`LLM content generator`

Data flow

Search logs feed an offline batch job that enriches each query with domain signals and fills prompt templates. The LLM generates substitute and complementary item concepts, which are stored by query, converted back through the existing product search engine, de-duplicated and reranked, then looked up at runtime alongside normal search results. A Whole Page Ranker chooses where to show the inspirational content.

3 supporting quotes

“A batch job extracts historical search queries from logs and enriches them with necessary metadata including QU signals, consecutive search terms, and other relevant signals.”summary, stated

“A batch job invokes the LLM and stores responses in a key-value store, with the query as key and the LLM response”summary, stated

“Each item in the LLM-generated list is treated as a search query and passed through Instacart's existing search engine to retrieve the best matching products from the catalog.”summary, stated

Why this architecture

Instacart precomputes LLM content because grocery search requires fast, predictable request-time serving. LLM world knowledge is improved by Instacart-specific query, conversion, and sequential-search signals, then existing search infrastructure maps concepts to actual products.

3 supporting quotes

“rather than serving LLM calls at request time with associated latency and cost concerns, Instacart pre-computes content offline for known queries.”summary, stated

“This fusion of LLM world knowledge with Instacart-specific context significantly improved recommendation accuracy.”summary, stated

“LLM outputs are treated as queries themselves to leverage existing product matching capabilities”summary, stated

Models used: LLM content generation model · Query Understanding model · Whole Page Ranker · LLM as a Judge evaluator

Scale

Instacart operates a four-sided marketplace.
“Instacart, a major grocery e-commerce platform operating a four-sided marketplace”summary, stated
Sequential-search analysis improved inspirational-content engagement by 18%.
“This methodology led to an 18% improvement in engagement rate with inspirational content.”summary, stated

Alternatives considered (roads not taken)

Use department-level related items

Rejected because: It returned loosely related products like cookies for croissant and missed complementary suggestions.

“for narrow queries like "croissant," it would return loosely related items like "cookies" simply because they shared a department category.”summary, stated

Use generic LLM world knowledge alone

Rejected because: Basic generation misread brand and category intent without Instacart-specific signals.

“basic generation often misinterpreted user intent or generated overly generic recommendations.”summary, stated

“Instacart augmented prompts with domain-specific signals:”summary, stated

Run LLM generation at request time

Rejected because: Offline generation traded real-time personalization for cost efficiency and consistent latency.

“This approach trades off real-time personalization for cost efficiency and consistent latency.”summary, stated

Patterns

`rag-retrieval` (recommendation)

Instacart augments LLM generation with search logs, conversion categories, query-understanding annotations, and sequential search terms.

“The prompt format for advanced generation explicitly includes annotations like ":BODYARMOR" for brand queries and ":pizza, :frozen" for attributed product searches”summary, stated

“By examining next converted search terms, the system provides richer context to the LLM. For "sour cream," instead of only considering sour cream products, the system incorporates data showing that users frequently purchase tortilla chips or baked potatoes afterward.”summary, stated

`batch-offline-processing` (serving)

LLM content is generated offline for known queries and served from a key-value store.

“The production system implements an offline batch processing architecture optimized for latency and cost:”summary, stated

“The key-value store serving model allows for rapid lookup during user sessions while the batch pipeline handles the computationally expensive LLM inference.”summary, stated

`retrieval-ranking` (ranking)

Whole Page Ranker and diversification logic decide where and how to show generated discovery content.

“They developed a "Whole Page Ranker" model that determines optimal positions for new content on the page. This model balances showing highly relevant content to users while maintaining revenue objectives, dynamically adjusting layout based on content type and relevance.”summary, stated

“Post-processing: The pipeline removes duplicates and similar products, filters irrelevant items, and applies diversity-based reranking to ensure users see varied options.”summary, stated

`llm-as-judge` (evaluation)

Instacart uses an LLM evaluator to score complementary and substitute recommendation content at scale.

“Instacart adopted the "LLM as a Judge" paradigm for quality evaluation. The evaluation prompt positions the LLM as an expert in e-commerce recommendation systems, tasking it to evaluate curator-generated content that produces complementary or substitute search terms.”summary, stated

“tasking it to evaluate curator-generated content that produces complementary or substitute search terms.”summary, stated

`query-rewriting` (search)

Generated concepts are treated as search queries and passed back through Instacart's existing search engine.

“Each item in the LLM-generated list is treated as a search query and passed through Instacart's existing search engine”summary, stated

Tradeoffs

Offline generation

Chose Precompute LLM content and store it by query over Generate discovery content during each search request.

Rationale: Precomputation controls latency and cost at the expense of real-time personalization.

“This approach trades off real-time personalization for cost efficiency and consistent latency. The key-value store serving model allows for rapid lookup during user sessions while the batch pipeline handles the computationally expensive LLM inference.”summary, stated

Domain-augmented prompts

Chose Fuse LLM world knowledge with Instacart query and conversion data over Use basic grocery-shopping prompts alone.

Rationale: Domain signals reduce misinterpretation and generic suggestions.

“This fusion of LLM world knowledge with Instacart-specific context significantly improved recommendation accuracy.”summary, stated

Reuse existing search for product mapping

Chose Treat LLM-generated items as search queries over Build a separate product retrieval system for generated recommendations.

Rationale: Existing search infrastructure already matched item concepts to catalog products.

“The integration with existing search infrastructure is notable—LLM outputs are treated as queries themselves to leverage existing product matching capabilities, avoiding the need to build entirely new retrieval systems.”summary, stated

Gotchas (what bit them)

Generic LLM recommendations can miss domain-specific intent.

Root cause: Brand names and grocery-specific conversion patterns are not reliably inferred from world knowledge alone.

Resolution: Instacart injects QU annotations, historical conversions, and purchase patterns into prompts.

“For instance, a search for "Just Mayo" (a vegan mayonnaise brand) would be misinterpreted as generic mayonnaise, and "protein" would return common protein sources rather than the protein bars and powders that users actually converted on.”summary, stated

“To address this, Instacart augmented prompts with domain-specific signals:”summary, stated

Offline LLM generation limits freshness.

Root cause: Content is precomputed and updated on a batch cadence rather than personalized at request time.

Resolution: The architecture accepts the freshness tradeoff for cost efficiency and consistent latency.

“The reliance on offline batch processing means content freshness is limited to daily updates.”summary, stated

Generated concepts can map to irrelevant products.

Root cause: LLM item concepts still need catalog grounding through search.

Resolution: Post-processing removes duplicates, filters irrelevant recalls, and diversifies results.

“The system also depends heavily on the quality of Query Understanding models for accurate intent annotation, and the product mapping step introduces potential for irrelevant recalls that require post-processing to filter.”summary, stated

Evaluation

Instacart evaluated discovery content with engagement metrics, revenue alignment, and an LLM-as-judge prompt that scores complementary/substitute content quality for e-commerce recommendations.

Metrics: 18% engagement-rate improvement · LLM quality scores · Revenue alignment · Content position through Whole Page Ranker

“The implementation mines frequently co-occurring lists of consecutive search terms to extract high-quality signals, filtering out noise from partial or varied shopping sessions. This methodology led to an 18% improvement in engagement rate with inspirational content.”summary, stated

“The goal is to assess whether the generated content would encourage users to make purchases, with the LLM providing quality scores.”summary, stated

Outcomes

Sequential search-term analysis improved inspirational-content engagement by 18%.
The system supports substitute and complementary product discovery.
Offline generation and key-value serving avoid request-time LLM latency.
Generated content is positioned with a Whole Page Ranker to balance relevance and revenue.

Instacart applies the system across grocery search queries in a four-sided marketplace, using batch generation and query-keyed serving for known searches.

“The results included substantial improvements in user engagement (18% improvement in engagement rate with inspirational content) and incremental revenue generation.”summary, stated

“When users issue queries, the system retrieves both standard search results and looks up the LLM-content table to display inspirational products”summary, stated