Hybrid AI System for Large-Scale Product Categorization

walmart · E-commerce · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: large-scale e-commerce product categorization and catalog relevance

Walmart needed to map product categories to the right product types across a very large online catalog so customers see relevant items, while avoiding the cost and latency of running generative models over millions of candidate pairs.

Constraints:

• Walmart.com has more than 400 million SKUs
• category-to-product-type relationships are many-to-many
• manual curation of millions of candidate category/product-type pairs is impractical
• GenAI inference is too costly to run on every candidate pair
• backend filtering must meet millisecond-range service-level requirements
• customer engagement data is noisy and can be misleading
• production edge cases still require exception handling and human intervention

Architecture

Walmart built Ghotok as a staged ensemble for product categorization. Predictive AI models trained on domain-specific, human-labeled features first score category/product-type pairs and apply confidence thresholds learned at a fixed false-positive rate, reducing millions of candidate pairs to thousands. Generative AI models then run only on this filtered set, using chain-of-thought prompts and hierarchy-aware symbol tuning to reason about semantic relevance and remove additional false positives. The resulting category-to-product-type mappings are integrated into the backend filtering path through a two-tier LRU cache that serves mappings from L1/L2 cache before falling back to primary storage. A separate exception-handling tool combines ML and human intervention for production edge cases.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Digital_storefront_category_tree["Digital storefront category tree"] --> Category_item_fetcher["Category item fetcher"]
  Digital_storefront_category_tree --> Ghotok_mapping_engine["Ghotok mapping engine"]
  Product_type_hierarchy["Product type hierarchy"] --> Ghotok_mapping_engine
  Human_labeled_training_set["Human-labeled training set"] --> Predictive_AI_models["Predictive AI models"]
  Human_labeled_training_set --> Validation_evaluator["Validation evaluator"]
  Predictive_AI_models --> Generative_AI_relevance_model["Generative AI relevance model"]
  Prompt_engineering_layer["Prompt engineering layer"] --> Generative_AI_relevance_model
  Generative_AI_relevance_model --> Ghotok_mapping_engine
  Predictive_AI_models --> Validation_evaluator
  Generative_AI_relevance_model --> Validation_evaluator
  Ghotok_mapping_engine --> Two_tier_LRU_cache["Two-tier LRU cache"]
  Two_tier_LRU_cache --> Primary_storage["Primary storage"]
  Two_tier_LRU_cache --> Backend_filtering_service["Backend filtering service"]
  Category_item_fetcher --> Backend_filtering_service
  Backend_filtering_service --> Digital_storefront_category_tree
  Ghotok_mapping_engine --> Exception_handling_tool["Exception handling tool"]

Components

Data flow

A user request fetches items by category while Ghotok identifies relevant product types for that category. Offline, predictive models learn thresholds over category/product-type candidate pairs from human-labeled data and filter millions of candidates down to thousands. The generative model then evaluates the remaining pairs with chain-of-thought prompting and full path representations. At serving time, the backend stores Category-to-ProductType mappings in L1 and L2 LRU caches, checks those caches before primary storage, and filters fetched items by the approved product types.

"Fetch items based on the category they fall into.

2. In parallel, identify the relevant product types for the category.

3. Filter the items fetched on step 1 based on the product types in step 2."

Why this architecture

Walmart chose a hybrid ensemble because predictive models are cheap enough to filter a massive candidate space, while generative models provide semantic understanding for the smaller set where that cost is justified. The constrained GenAI role reduces hallucination risk by using the model to eliminate false positives rather than invent content. The two-tier cache is necessary because the offline mapping data can contain millions of rows, but the backend needs responses in a few milliseconds.

Models used: predictive AI models · generative AI models · ensemble models · million-parameter predictive models · billion-parameter generative models

Scale

• Walmart.com has more than 400 million SKUs.
  “With over 400 million SKUs, Walmart.com must streamline the online shopping process to enhance customer satisfaction.”article, stated
• The offline category/product-type data can reach several million rows.
  “the resulting data set can lead to several million rows of offline data.”article, stated
• Generative AI is run after predictive filtering reduces candidates from millions to thousands.
  “This reduces the total number of candidate pairs to select from millions to thousands.”article, stated
• The backend SLA is in the few-millisecond range.
  “typical service level agreement requirements that usually range in a few milliseconds”article, stated
• The catalog has several thousand categories, each connected to hundreds of product-type nodes.
  “Our product catalog involves several thousand categories. Each of these categories is connected to, or can correlate with, hundreds of different ProductType nodes.”article, stated

Alternatives considered (roads not taken)

Run Generative AI on all candidate pairs

Rejected because: Generative inference was too costly at the millions-of-pairs scale, so predictive filtering runs first.

Use one model family for the entire task

Rejected because: The team explicitly uses an ensemble of predictive and generative models to combine cheaper filtering with stronger semantic understanding.

Train on noisy customer engagement data

Rejected because: Walmart uses limited human-labeled data because clicks can be accidental or curiosity-driven.

Use only category node names in the prompt

Rejected because: The team found that full root-to-node path representation and leaf-node emphasis improved reasoning quality.

Query primary storage for every request

Rejected because: The backend needs millisecond response times over millions of entries, so L1/L2 LRU caches serve mappings before storage is queried.

Patterns

cost-aware-routing (operations)

Predictive models cheaply reduce the candidate set before the expensive generative model is invoked.

confidence-thresholding (quality)

The predictive stage learns thresholds at a fixed false-positive rate and passes only higher-confidence candidate pairs to GenAI.

structured-output-contract (data-modeling)

The system is organized around a structured mapping from Category to ProductType sets, which is cached and used to filter catalog items.

latency-optimization (operations)

A two-tier LRU cache keeps category/product-type mappings available within millisecond backend SLA requirements.

human-in-the-loop-gate (oversight)

An exception-handling tool uses human intervention for edge cases that the predictive/generative ensemble cannot resolve cleanly.

eval-driven-development (quality)

Human-labeled validation data and metrics such as precision, recall, F1, TPR, and FPR guide hyperparameters, thresholds, and ensemble comparisons.

Tradeoffs

Predictive filtering before generative reasoning

Chose cheap predictive models to narrow candidate pairs, then GenAI on the reduced set over running GenAI on all millions of candidate pairs.

Rationale: This preserves semantic reasoning where it matters while controlling inference cost at catalog scale.

Human-labeled data instead of click data

Chose limited human-labeled data over customer engagement data.

Rationale: Click data is noisy, while labeled data supports both frequent and rarely visited hierarchy regions.

Constrain GenAI to false-positive elimination

Chose semantic comprehension over pre-filtered pairs over open-ended generation of product classifications.

Rationale: The constrained role lets Walmart benefit from GenAI semantics while reducing hallucination exposure.

Full path representation for prompts

Chose root-to-node path string with leaf-node emphasis over context node names only.

Rationale: The full hierarchy path gave the model better semantic context for relevance decisions.

Two-level cache for serving mappings

Chose L1 and L2 LRU caches before primary storage over direct primary-storage lookup for each filtering request.

Rationale: The backend must process millions of entries with service-level expectations in a few milliseconds.

Gotchas (what bit them)

Generative inference is too expensive for the raw candidate space.

Root cause: The catalog has millions of candidate category/product-type pairs, and GenAI inference is costly.

Resolution: Use predictive AI thresholds to reduce candidates from millions to thousands before applying generative AI.

Customer engagement data can be a noisy training signal.

Root cause: Customers may click items accidentally or out of curiosity, so click behavior does not reliably indicate category relevance.

Resolution: Use a limited amount of human-labeled data for training and evaluation instead of relying on engagement logs.

Hierarchy representation matters for LLM relevance reasoning.

Root cause: Using only node names loses tree context needed to understand how a product type relates to a category path.

Resolution: Pass the full root-to-node path string and instruct the model to weight the leaf node more heavily.

The production backend cannot scan offline mappings on request.

Root cause: Several thousand categories times hundreds of product types produce millions of offline rows, but the service needs millisecond responses.

Resolution: Use L1 and L2 LRU caches for Category-to-ProductType mappings, querying primary storage only on cache miss.

Automated categorization still encounters production edge cases.

Root cause: Even with lower false-positive rates, deployed systems see cases outside the model's reliable operating envelope.

Resolution: Deploy an exception-handling tool powered by ML and human intervention.

Evaluation

Walmart uses human-labeled data to tune predictive-model hyperparameters, set confidence thresholds at a fixed false-positive rate, compare precision/recall/F1/TPR/FPR metrics, and validate that the predictive-plus-generative ensemble performs best on a validation set.

Metrics: precision · recall · F1 · true positive rate · false positive rate · fixed FPR confidence thresholds · validation-set ensemble performance

Outcomes

• Ghotok maps relationships between Walmart categories and product types to improve product filtering.
• Predictive filtering reduces millions of candidate pairs to thousands before GenAI is used.
• Chain-of-thought prompting improved adherence to system prompts and output quality.
• Full path representation plus leaf-node emphasis improved generated reasoning.
• Two-tier caching supports millisecond-range backend serving over millions of offline mappings.

The article is unusually concrete about production scale: 400 million SKUs, several thousand categories, hundreds of product types, millions of offline rows, and few-millisecond SLA expectations. It does not disclose exact precision/recall numbers for the final ensemble.