Semantic Caching for E-commerce Search Optimization

walmart · E-commerce · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce search relevance and latency optimization

Walmart needed search to understand natural-language shopping intent and reduce expensive generative search work without relying on exact-match caching alone.

Constraints:

• Traditional caches only help when the incoming query exactly matches a cached key.
• Customer queries vary semantically even when they ask for similar products or occasions.
• Semantic vector search adds storage, compute, and latency costs compared with simple caching.
• Search responses still need to move toward sub-second latency for customer-facing commerce.

Architecture

Walmart layered semantic caching and generative search over e-commerce search: customer queries are interpreted by intent, semantically similar tail queries can hit cached responses through embeddings/vector search, and exact plus semantic caching are combined to manage the latency and compute tradeoff while returning broader product groupings such as snacks, drinks, apparel, and televisions for occasion-style queries.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Customer search query"] --> B["Exact-match cache"]
  A --> C["Semantic cache"]
  C --> D["Vector search layer"]
  D --> E["Product catalog"]
  C --> F["Generative search layer"]
  E --> F
  B --> G["Search results"]
  C --> G
  F --> G

Components

Data flow

A shopper query enters search, exact cache handles literal repeats, semantic cache embeds or compares query meaning against stored semantic results, generative search expands intent into relevant product groupings, and the system balances semantic-cache recall against the latency and compute cost of vectorizing and searching products and queries.

Why this architecture

The architecture keeps exact caching for cheap repeat traffic while adding semantic matching for tail-query variations that exact matching cannot reuse, because Walmart wanted relevance gains without letting vector search latency and cost dominate every request.

Models used: embeddings · generative AI models

Scale

• Semantic caching reached close to a 50% cache rate for tail queries after Walmart expected only 10-20%.
  “What we anticipated might yield a 10-20% caching rate for tail queries pleasantly surprised us with a rate closer to 50%”article, stated
• The system was designed for sub-second customer-facing search responses.
  “We are progressing towards achieving sub-second response times, ensuring that our customers receive timely and relevant results”article, stated
• The implementation must vectorize and search every product SKU and query for semantic caching.
  “semantic caching needs every product SKU and query to be converted to vectors and then searched upon”article, stated

Alternatives considered (roads not taken)

Use exact-match caching only

Rejected because: Exact-match caching misses natural-language variations of the same shopping intent.

Use semantic caching alone for broader recall

Rejected because: Vector conversion and semantic search introduce latency and compute costs that make a combined simple-plus-semantic cache more practical.

Patterns

semantic-caching (performance)

Walmart caches semantically equivalent tail queries by meaning instead of requiring exact query-string matches.

cost-aware-routing (optimization)

The system combines cheap simple caching with more expensive semantic/vector caching so not every query pays the semantic path's cost.

latency-optimization (performance)

Walmart frames semantic caching around the customer-facing need to progress toward sub-second search responses.

Tradeoffs

Combine simple and semantic caching.

Chose simple cache plus semantic cache over semantic vector cache as the only cache layer.

Rationale: Simple caching remains cheap and fast, while semantic caching expands recall for language variations.

Accept vector-search cost for tail-query reuse.

Chose embedding/vector search for semantic equivalence over low-cost exact cache with lower tail coverage.

Rationale: The realized tail-query cache rate was much higher than the expected rate, making the added infrastructure valuable.

Gotchas (what bit them)

Exact matching under-caches natural-language tail queries.

Root cause: Customers express the same need in many phrasings, but exact caches treat each phrase as a different key.

Resolution: Use semantic caching that interprets query meaning rather than literal strings.

Semantic caching can slow the search path.

Root cause: Semantic cache lookup requires more computation than a simple cache.

Resolution: Balance the semantic path with performance engineering toward sub-second response times.

Vector-based cache infrastructure can be expensive.

Root cause: The semantic cache has to convert and search product SKUs and queries as vectors.

Resolution: Use semantic caching selectively alongside simple caching rather than replacing the cheap path wholesale.

Evaluation

The public writeup evaluates semantic caching through cache-hit expectations versus observed tail-query caching, qualitative examples of broader intent understanding, and latency/cost tradeoff analysis.

Metrics: tail-query cache rate · expected cache rate · response latency · storage and compute cost · search-result relevance

Outcomes

• Semantic caching reportedly handled tail queries at close to a 50% cache rate.
• Generative search could return broader product groupings for intent-based queries such as football watch parties.
• The architecture exposed an explicit cost-performance tradeoff that favored combining simple and semantic caches.

The public source gives cache-rate and latency targets but does not disclose total query volume.