Product Attribute Normalization and Sorting Using DSPy for Large-Scale E-commerce

zoro_uk · E-commerce · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce product attribute normalization and sorting

Zoro UK needed to normalize and logically sort product attributes across a massive supplier-driven catalog so customers could search and filter products in meaningful order instead of seeing inconsistent strings and units.

Constraints:

• The catalog contains millions of products and tens of thousands of attribute types.
• More than 300 suppliers represent the same product measurements in inconsistent formats.
• Manual normalization is not feasible at the scale of 75,000 product attributes.
• Some attributes can be sorted alphanumerically after normalization, while others require semantic ordering such as clothing sizes.
• Using expensive models for every attribute would waste cost when simpler sorting or smaller models are sufficient.

Architecture

Zoro built a DSPy-backed product attribute sorting pipeline. A scheduled job pulls attribute values from MongoDB for a given attribute type and sends them to an NLM Sort Service. The service first uses a smaller model, described as Mistral 8B in the summary, to decide whether normal alphanumeric sorting is enough. If it is, Python performs the sort; if not, the service routes the values to GPT-4 for semantic/logical ordering. DSPy handles prompt and example optimization around task-specific metrics, allowing the team to switch models and compile optimized prompts without manual prompt engineering.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  mongodb-product-catalog["MongoDB product catalog"] --> scheduled-sorting-job["Scheduled sorting job"]
  scheduled-sorting-job["Scheduled sorting job"] --> mongodb-product-catalog["MongoDB product catalog"]
  scheduled-sorting-job["Scheduled sorting job"] --> nlm-sort-service["NLM Sort Service"]
  nlm-sort-service["NLM Sort Service"] --> scheduled-sorting-job["Scheduled sorting job"]
  nlm-sort-service["NLM Sort Service"] --> mistral-classifier["Mistral classifier"]
  mistral-classifier["Mistral classifier"] --> nlm-sort-service["NLM Sort Service"]
  nlm-sort-service["NLM Sort Service"] --> python-alphanumeric-sorter["Python alphanumeric sorter"]
  python-alphanumeric-sorter["Python alphanumeric sorter"] --> nlm-sort-service["NLM Sort Service"]
  python-alphanumeric-sorter["Python alphanumeric sorter"] --> mongodb-product-catalog["MongoDB product catalog"]
  mongodb-product-catalog["MongoDB product catalog"] --> python-alphanumeric-sorter["Python alphanumeric sorter"]
  nlm-sort-service["NLM Sort Service"] --> gpt-4-semantic-sorter["GPT-4 semantic sorter"]
  gpt-4-semantic-sorter["GPT-4 semantic sorter"] --> nlm-sort-service["NLM Sort Service"]
  gpt-4-semantic-sorter["GPT-4 semantic sorter"] --> mongodb-product-catalog["MongoDB product catalog"]
  mongodb-product-catalog["MongoDB product catalog"] --> gpt-4-semantic-sorter["GPT-4 semantic sorter"]
  dspy-optimizer["DSPy optimizer"] --> mistral-classifier["Mistral classifier"]
  mistral-classifier["Mistral classifier"] --> dspy-optimizer["DSPy optimizer"]
  dspy-optimizer["DSPy optimizer"] --> gpt-4-semantic-sorter["GPT-4 semantic sorter"]
  gpt-4-semantic-sorter["GPT-4 semantic sorter"] --> dspy-optimizer["DSPy optimizer"]
  mongodb-product-catalog["MongoDB product catalog"] --> sorted-search-experience["Sorted search experience"]
  sorted-search-experience["Sorted search experience"] --> mongodb-product-catalog["MongoDB product catalog"]

Components

Data flow

Product and attribute data live in MongoDB. A scheduled sorting job gathers all values for a particular attribute, such as function length, across products. The NLM Sort Service classifies whether the values can be handled by alphanumeric sorting. Straightforward numeric/unit cases go through Python sorting, while semantically ordered values such as clothing sizes are routed to GPT-4. The sorted values are written back to MongoDB and served to the e-commerce search/filter experience.

Why this architecture

The architecture separates cheap classification and deterministic sorting from expensive semantic sorting because most catalog values should not require GPT-4. DSPy was used because it lets the team define metrics and examples, compile optimized prompts, and switch models without hand-maintaining long prompts.

Models used: Mistral 8B · GPT-4 · DSPy MIPRO optimizer

Scale

• Zoro UK had roughly 1 million monthly active users.
  “right now we are doing about uh 1 million monthly active users on our e-commerce site”transcript, stated
• The company sells about 3.5 million products.
  “in our company we actually have uh about 3 and a half million products that we sell”transcript, stated
• The catalog comes from more than 300 suppliers.
  “we actually have um more than 300 plus suppliers and they all Supply similar kind of stuff”transcript, stated
• The site has more than 75,000 product attributes.
  “our site we actually have more than 75 different 75,000 different attributes”transcript, stated
• The example function-length attribute applied to about 2,500 products.
  “function length uh for some particular uh set of products uh and we have 2 and a half thousand is products with that kind of thing”transcript, stated
• The Sanskrit learning example improved from 51.7% zero-shot to 70.0% few-shot and 71.5% with MIPRO under a limited 10-iteration run.
  “the zero shot uh 51.7% U with few shot itself it actually got a pretty big bump which is 70.0”transcript, stated

  “I just gave it like 10 iterations or so so that's when it just gave me this one and a half percentage uh bump”transcript, stated

Alternatives considered (roads not taken)

Manual product attribute normalization

Rejected because: The supplier and attribute scale made manual normalization humanly impractical.

Hand-written long prompt engineering

Rejected because: DSPy automatically optimizes prompts and examples around metrics, removing the need to maintain long prompts manually.

Use one model for every case

Rejected because: The team used a smaller model first and routed only semantic sorting cases to GPT-4, keeping simpler cases deterministic or cheaper.

Always semantic-sort with GPT-4

Rejected because: When alphanumeric sorting is enough, Python can sort without a model.

Patterns

model-routing (architecture)

The sort service routes attributes through a smaller model for classification and only sends semantic cases to GPT-4.

cost-aware-routing (operations)

The architecture reserves GPT-4 for cases that need semantic reasoning and uses Python or a smaller model where sufficient.

eval-driven-development (quality)

DSPy development centers on defining metrics, running baseline evaluation, compiling an optimizer, and iterating with better metrics and examples.

prompt-versioning (prompt-operations)

DSPy compiles and optimizes prompt/example choices instead of relying on manual prompt editing.

structured-output-contract (data-quality)

The system turns messy supplier attribute strings into a normalized sorted order that can be stored and served consistently.

data-quality-loop (catalog-operations)

The pipeline repeatedly pulls product attribute values, normalizes/sorts them, writes them back, and improves the search/filter experience.

Tradeoffs

Use a two-tier model pipeline for sorting.

Chose Small-model classification plus GPT-4 only for semantic sorting over Sending every attribute directly to GPT-4.

Rationale: The smaller model can identify easy alphanumeric cases and reserve expensive semantic reasoning for values that need it.

Use deterministic Python sorting when alphanumeric order is sufficient.

Chose Python-based alphanumeric sorting over Model-based sorting for every attribute.

Rationale: Ordinary deterministic sorting is enough for simple normalized numeric or string cases.

Adopt DSPy to optimize prompts programmatically.

Chose Metric-driven DSPy compilation over Manual long-prompt tuning.

Rationale: DSPy let the team optimize prompts/examples, remain LLM-agnostic, and switch from one model to another with recompilation.

Limit optimization iterations for the learning example.

Chose A small 10-iteration MIPRO run over A much larger optimization search.

Rationale: The prompt optimization calls the LLM repeatedly, so cost limited experimentation.

Gotchas (what bit them)

Supplier data represented the same attribute in incompatible formats.

Root cause: Hundreds of suppliers encoded dimensions and units differently, so identical concepts appeared as mixed strings, split values, or imperial units.

Resolution: The sorting pipeline aggregates each attribute type and normalizes/sorts values before serving them back to search.

A metric mistake would undermine the whole DSPy optimization.

Root cause: DSPy optimizes toward the metric, so the quality of the metric directly determines the quality of the optimized pipeline.

Resolution: The speaker emphasizes metric definition, baseline evaluation, and iteration before relying on optimized prompts.

Alphanumeric sorting fails for logical categories like clothing sizes.

Root cause: Values such as S, M, L, and XL have a semantic order that ordinary string sorting does not capture.

Resolution: The Mistral classifier sends non-alphanumeric cases to GPT-4 for semantic sorting.

Optimization experiments can become expensive because each iteration calls an LLM.

Root cause: DSPy prompt optimization itself invokes models during compilation/evaluation.

Resolution: The speaker limited MIPRO to about 10 iterations in the Sanskrit experiment because of computational cost.

Evaluation

Zoro's DSPy approach uses explicitly defined task metrics, zero-shot baselines, few-shot comparisons, optimizer compilation, and iteration. The production sorter is evaluated by whether it can classify alphanumeric versus semantic sorting and produce usable sorted attributes for search.

Metrics: Zero-shot Sanskrit translation score of 51.7% · Few-shot Sanskrit translation score of 70.0% · MIPRO score of about 71.5% after roughly 10 iterations · Product catalog scale of 3.5 million products and 75,000 attributes · Attribute-specific correctness for alphanumeric versus semantic sorting

Outcomes

• Zoro built a production attribute sorting pipeline over MongoDB-backed catalog data.
• DSPy eliminated much of the hand prompt-engineering burden and let the team switch between models with recompilation.
• The two-tier architecture routed simple cases to smaller/deterministic paths and complex semantic cases to GPT-4.
• The sorted attributes improved product discoverability and filtering for a large e-commerce catalog.

The system supports a 3.5M-product catalog, 300+ suppliers, 75,000 attribute types, about 1M monthly active users, and examples such as 2,500 products sharing one function-length attribute.