Automated Product Classification and Attribute Extraction Using Vision LLMs

shopify · E-commerce · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce product taxonomy and enrichment

Shopify needed to infer reliable product categories and attributes from flexible merchant listings at very large scale so downstream taxes, search, recommendations, compliance, and merchandising systems could use structured product data.

Constraints:

• merchant product types contained tens of millions of unique free-text values
• forcing merchants to fill standardized taxonomy fields added too much friction
• tax and legal use cases make egregious classification errors expensive
• the system needs both low-latency UI predictions and high-throughput batch/API predictions
• fine-tuning data must cover long-tail categories rather than only the most common products

Architecture

Shopify first built a controlled product taxonomy with about 12,000 categories and finite attribute values. The classification pipeline then uses a two-step vision-LLM process: first, image and product text produce a category plus product/image metadata; second, the predicted category determines which controlled attributes are injected into a prompt for extraction. Hallucinated categories are corrected post hoc through nearest real-category lookup rather than constrained grammar. To make the system affordable at tens of millions of daily predictions, Shopify builds synthetic golden datasets with GPT/Claude/Gemini agreement and human validation, fine-tunes smaller models, evaluates with Ray/SGLang over about 2 million records, and serves through both low-latency Kubernetes/LM Deploy paths and high-throughput Kafka/Dataflow/Triton-style batching.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  merchant-product-listing[Merchant product listing] --> category-prediction-model[Category prediction model]
  shopify-product-taxonomy[Shopify product taxonomy] --> category-prediction-model
  category-prediction-model --> post-hoc-taxonomy-matcher[Post-hoc taxonomy matcher]
  post-hoc-taxonomy-matcher --> shopify-product-taxonomy
  category-prediction-model --> attribute-extraction-model[Attribute extraction model]
  shopify-product-taxonomy --> attribute-extraction-model
  synthetic-label-pipeline[Synthetic label pipeline] --> fine-tuning-infrastructure[Fine-tuning infrastructure]
  fine-tuning-infrastructure --> evaluation-pipeline[Evaluation pipeline]
  attribute-extraction-model --> real-time-serving-path[Real-time serving path]
  attribute-extraction-model --> high-throughput-batch-path[High-throughput batch path]
  real-time-serving-path --> merchant-feedback-queue[Merchant feedback queue]
  merchant-feedback-queue --> synthetic-label-pipeline

Components

Data flow

Merchant product images, titles, and descriptions enter a vision-LLM category predictor. The category is validated against Shopify's taxonomy and used to select relevant controlled attributes. A second LLM call extracts category-specific attributes into structured JSON. Synthetic labels from multiple frontier models plus human annotators create fine-tuning data, while merchant corrections and human/LLM vetting feed ongoing evaluation. Predictions are served either directly in the admin UI or through high-throughput Kafka/Dataflow batch pipelines for API and bulk-created products.

Why this architecture

Shopify split category prediction from attribute extraction because attributes are category-specific; a t-shirt and a screwdriver need different prompts and value sets. The taxonomy constrains a previously unbounded merchant vocabulary into about 12,000 categories and finite attribute values. Fine-tuning smaller models is necessary because frontier vision LLMs are too expensive for tens of millions of daily predictions. The serving architecture splits real-time UI predictions from high-throughput batch/API inference because each path optimizes for different latency and batching needs.

Models used: vision LLMs · GPT · Claude · Gemini · fine-tuned smaller models · multilayer perceptron baseline

Scale

• Merchant-provided product-type values contained tens of millions of unique strings.
  “looked at that product type and there are tens of millions of unique things that are in that product type right so”transcript, stated
• Shopify's taxonomy includes about 12,000 categories.
  “right like there's like 12,000 or so categories in our system usually it's like 50 or so categories that dominate all sales and”transcript, stated
• The system runs tens of millions of predictions every day.
  “yeah there's tens of millions of predictions every day right uh and I already gave this away in the start this”transcript, stated
• The evaluation dataset is about 2 million records.
  “gnarly because our evaluation data set is about 2 million records so even just evaluating something can be a little”transcript, stated
• The core productionization team was about four people for three months after the one-person prototype.
  “like the entire prototype was like just me playing around and kind of going what can I do with it once you get to a place”transcript, stated

  “then we got I would say four people working on this for 3 months and we did have sister teams we have like an infra”transcript, stated

Alternatives considered (roads not taken)

Ask merchants to select standardized categories

Rejected because: Merchants ignored the request because it added workflow friction and did not directly help them.

Use real-world distribution for training data

Rejected because: Dominant categories would hide poor long-tail performance, so Shopify intentionally enforces a uniform training distribution.

Constrained grammar for taxonomy-valid categories

Rejected because: Grammar enforcement hurt more than it helped, so the team allowed hallucinations and corrected them with post-hoc lookup.

Use GPT-4-class models directly for all predictions

Rejected because: Frontier model quality was attractive but cost was too high for production volume.

Patterns

structured-output-contract (output-control)

The model emits category and attribute predictions against controlled taxonomy values as structured JSON.

synthetic-data-generation (data)

Shopify uses GPT, Claude, Gemini, and LLM agreement to generate category and attribute labels before human validation.

fine-tuning-adaptation (model-operations)

The team fine-tunes smaller models from synthetic/human-validated datasets to manage cost and quality at scale.

cost-aware-routing (operations)

Shopify adds caching, trigger logic, and separate low-latency/high-throughput inference paths to reduce production cost.

human-in-the-loop-gate (governance)

Human annotators validate synthetic data and merchant feedback before it becomes training signal.

Tradeoffs

Use a two-call category-then-attribute pipeline

Chose category prediction followed by category-specific attribute extraction over one prompt that extracts every possible attribute at once.

Rationale: Only the known category tells the model which controlled attributes and values are relevant.

Fix hallucinated categories after inference

Chose post-hoc nearest taxonomy lookup over grammar-constrained decoding.

Rationale: Constrained grammar hurt performance, while post-hoc correction handled rare invalid category outputs.

Separate real-time and high-throughput serving paths

Chose low-latency UI serving plus Kafka/Dataflow batch inference over one inference path for all product creation modes.

Rationale: Merchant UI changes need quick responses, while API/bulk creation is throughput-sensitive and benefits from large batches.

Do not automatically train on all merchant feedback

Chose feedback queue with human and LLM vetting over direct merchant-feedback ingestion into training.

Rationale: Some corrections are contextual or subjective, and some product-category mistakes have legal or safety consequences.

Gotchas (what bit them)

Merchant-provided product types are too unconstrained for ML targets.

Root cause: Merchants can name and organize products however they want, creating tens of millions of unique product-type values.

Resolution: Build and govern a product taxonomy with controlled categories, attributes, and finite values.

Vision LLMs can hallucinate invalid taxonomy categories.

Root cause: The model can output a plausible category string that does not exist in the controlled taxonomy.

Resolution: Apply a post-hoc phase/index lookup to map outputs to the closest real category; this occurs less than 2% overall.

Merchant feedback is not always ground truth.

Root cause: Some attributes such as color or regulated product categories are subjective or high-stakes, so merchant corrections can be ambiguous.

Resolution: Queue feedback for human and LLM vetting instead of automatically training on every merchant correction.

Training distribution and serving distribution intentionally diverge.

Root cause: Real-world product volume is dominated by about 50 common categories, but optimizing only for those would hurt long-tail category quality.

Resolution: Train on an artificially uniform category distribution while accepting serving skew in production.

Evaluation

Shopify evaluates with large offline datasets, hierarchical precision/recall, baselines against an existing neural-network classifier, merchant accept/correct/delete signals, and separate continuous human annotation.

Metrics: about 2 million evaluation records · precision and recall · hierarchical precision and recall · merchant correction/accept/delete signals · human-annotator evaluation · long-tail category performance under uniform training distribution

Outcomes

• The vision-LLM approach improved over the prior classical neural-network baseline in the reported evaluation.
• The system supports merchant-facing real-time predictions and high-throughput batch/API predictions.
• Predictions are used by Shopify search, recommendations, taxes, and legal-compliance systems.
• Merchant feedback is collected but vetted before becoming model-training signal.

The system handles tens of millions of daily predictions across a taxonomy of about 12,000 categories, with evaluation over about 2 million records.