LLM-Powered Product Catalogue Quality Control at Scale

amazon · E-commerce · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Product catalogue quality control

Amazon needed to keep product listing data complete, accurate, and appealing across a huge catalogue where specialized category models and manual review could not cover nuanced attributes, new values, languages, and long-tail product types economically.

Constraints:

Architecture

Amazon adapted general-purpose LLMs to catalogue quality control through prompt tuning and catalogue-knowledge construction. The team organized the catalogue by product type and attribute value, used statistics about value frequency and customer views to identify likely standard values, exposed LLMs to schemas, rules, and terminology, and deployed the tuned LLM across the catalogue to recognize standard values, collect synonyms, and detect erroneous data.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Knowledge["Catalogue knowledge builder"] --> Tuning["Prompt tuning loop"]
  Tuning --> LLM["Catalogue quality LLM"]
  LLM --> Workflow["Catalogue quality workflow"]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Catalogue knowledge builder Organizes product types and attribute values and computes statistics about how values appear across the catalogue. Prompt tuning loop
Prompt tuning loop Exposes general-purpose LLMs to catalogue schemas, rules, terms, granularity constraints, and reasoning prompts. Catalogue knowledge builder, Catalogue quality LLM
Catalogue quality LLM Recognizes standard values, collects synonyms, and detects erroneous or nonsensical listing data. Prompt tuning loop, Catalogue quality workflow
Catalogue quality workflow Applies LLM quality-control outputs across product listings, languages, and previously cost-prohibitive catalogue areas. Catalogue quality LLM

Data flow

Amazon summarizes and organizes catalogue data by product type and attribute value, uses value statistics as signals of correctness, injects catalogue schemas and rules through prompt tuning, asks the LLM for reasoning to improve and debug prompts, and then exposes the LLM to the entire catalogue for quality-control tasks.

3 supporting quotes
“That process starts with summarizing and organizing the entire catalogue by product type and attribute value, similar, in some ways, to grouping the rows of a very large, very complex spreadsheet.”article, stated
“Through this reorganization, we can see the range of seller-provided attribute values for various product types and, importantly, the statistics on how often and where those values appear.”article, stated
“After many rounds of prompt tuning, the LLM is ready to be exposed to the entire catalogue, where it performs three main tasks”article, stated

Why this architecture

Amazon chose adaptable LLMs because independent specialized ML models could not economically cover complicated attributes, nuanced values, new languages, and long-tail catalogue areas. Prompt tuning let the team adapt general-purpose LLMs to catalogue vocabulary and rules without building a separate model for every product category.

3 supporting quotes
“there are many products in the catalogue with attributes that are much more complicated or nuanced that require specially trained ML models or manual review.”article, stated
“we’ve turned to more adaptable and generalizable large language models (LLMs).”article, stated
“When prompted with attribute data from the catalogue, LLMs adapt to the catalogue structures and vocabulary, allowing them to be usefully integrated into the quality control process.”article, stated

Models used: general-purpose LLMs · LLMs

Scale

Alternatives considered (roads not taken)

Independent specialized ML models per product category

Rejected because: They worked for simpler structured attributes but required special models or manual review for complicated or nuanced attributes.

“Amazon has traditionally relied on specialized machine learning (ML) models, each optimized for an independent product category, from patio furniture to headphones.”article, stated

Manual review for nuanced catalogue attributes

Rejected because: Manual review was too costly to scale across the catalogue and the LLM process saved thousands of review hours.

“there are many products in the catalogue with attributes that are much more complicated or nuanced that require specially trained ML models or manual review.”article, stated

Frequency-only attribute normalization

Rejected because: The most common value can be less informative than a more granular attribute value.

“even though “stainless steel” is a more likely attribute value, we don’t want to eliminate “440 stainless steel.””article, stated

Patterns

context-engineering (prompting)

Amazon builds catalogue knowledge from product types, attribute values, statistics, schemas, rules, and terminology, then exposes that context to the LLM through prompt tuning.

“we needed to systematically introduce it to the attribute semantics and values that most accurately describe millions of products and product types.”article, stated

structured-output-contract (data-quality)

The LLM is constrained to return values at the granularity of candidate lists and to perform specific catalogue quality tasks.

“we might prompt it with the phrase “The values returned must match the granularity, or broadness, of the values in the candidate list.””article, stated

eval-driven-development (prompt-iteration)

Amazon asks the LLM for reasoning so engineers can understand decisions and continue refining prompts.

“We can also ask an LLM for the reasoning behind its response, since this tends to improve its performance, as well as giving engineers insights that help them further fine-tune their prompts.”article, stated

Tradeoffs

Adopt adaptable LLMs for catalogue quality

Chose generalizable LLMs prompted with catalogue data over separate specialized ML models for each nuanced category.

Rationale: The LLM approach extends quality control into more categories and languages where specialized models were cost-prohibitive.

“Our LLM-based method allows us to extend the quality control process into the furthest reaches of the catalogue, which would have been cost prohibitive to explore with our prior process.”article, stated

Preserve attribute granularity

Chose prompted granularity matching over collapsing to the most frequent standard value.

Rationale: Granular values such as 440 stainless steel can be more informative than broader common values.

“The second is more granular; even though “stainless steel” is a more likely attribute value, we don’t want to eliminate “440 stainless steel.””article, stated

Gotchas (what bit them)

Common catalogue values are not always the best values.

Root cause: Frequency statistics can favor broad values even when a more granular value carries useful product information.

Resolution: Prompt the LLM to preserve the granularity of candidate values.

“While attribute statistics work well for many attributes, they won’t work for all of them, especially when there’s more nuance involved.”article, stated

Specialized category models do not cover the long tail economically.

Root cause: Complicated and nuanced product attributes require special models or manual review.

Resolution: Use adaptable LLMs prompted with catalogue structures and vocabulary.

“there are many products in the catalogue with attributes that are much more complicated or nuanced that require specially trained ML models or manual review.”article, stated

Prompt tuning needs transparent model reasoning for refinement.

Root cause: Engineers need insight into why the LLM selected a value to improve prompts.

Resolution: Ask the LLM for reasoning behind responses during tuning.

“giving engineers insights that help them further fine-tune their prompts.”article, stated

Evaluation

The article describes iterative prompt tuning, catalogue statistics, and LLM reasoning as engineering feedback mechanisms, but does not publish quantitative model-accuracy metrics or false-positive rates.

Metrics: hundreds of millions of products · millions of daily listing additions and edits · latest seller values included within days · thousands of human-review hours saved

“These statistics are fairly good indicators of a value’s correctness.”article, stated
“We can also ask an LLM for the reasoning behind its response, since this tends to improve its performance”article, stated

Outcomes

The public article gives strong scale context but limited accuracy detail: hundreds of millions of products, millions of daily listing changes, updates within days, and thousands of review hours saved.

3 supporting quotes
“where it performs three main tasks: recognizing standard attribute values, to establish correctness; collecting alternative representations of standard values, or synonyms; and detecting erroneous or nonsensical data entries.”article, stated
“The new process ensures that the latest seller values are included in the catalogue more quickly (within days) and saves thousands of hours in human reviews.”article, stated
“we’ve been able to use the LLM to increase the number of languages we can monitor and update.”article, stated