Vision Language Models for Large-Scale Product Classification and Understanding

shopify · E-commerce · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce product understanding and classification

Shopify needed to classify and enrich diverse merchant products at massive scale so search, recommendations, tax, content safety, and catalog organization could use consistent category and attribute data.

Constraints:

• The platform processes more than 30 million predictions daily.
• The taxonomy spans more than 10,000 categories and more than 1,000 product attributes.
• The model stack must balance classification quality with GPU cost and throughput.
• Category and attribute predictions must succeed consistently together.
• Merchant acceptance remains imperfect, so millions of daily predictions may still require intervention.

Architecture

Shopify built a near-real-time two-stage Vision Language Model pipeline on Dataflow and Kubernetes. Product data is validated and batched, a Vision LM first predicts category and simplified description against the Shopify Product Taxonomy, a second call predicts category-specific attributes, consistency management retries partial failures, and output processing validates results against taxonomy rules before downstream product surfaces consume them.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  product-input-stream["Product input stream"] --> dataflow-pipeline["Dataflow pipeline"]
  dataflow-pipeline --> vision-lm-service["Vision LM service"]
  vision-lm-service --> shopify-product-taxonomy["Shopify Product Taxonomy"]
  vision-lm-service --> dynamo-serving-stack["Dynamo serving stack"]
  dynamo-serving-stack --> kubernetes-gpu-cluster["Kubernetes GPU cluster"]
  dataflow-pipeline --> consistency-manager["Consistency manager"]
  consistency-manager --> output-processor["Output processor"]
  shopify-product-taxonomy --> output-processor
  output-processor --> downstream-commerce-surfaces["Downstream commerce surfaces"]

Components

Data flow

Product records enter a Dataflow pipeline, are dynamically batched and validated, pass through a Vision LM service on Kubernetes GPUs, receive category predictions first, then attribute predictions using category context, and finally undergo taxonomy validation and storage for search, recommendations, tax, and trust-and-safety use cases.

Why this architecture

The two-stage VLM architecture matches Shopify's taxonomy problem: category prediction must establish context before attributes can be extracted consistently, while FP8 quantization, in-flight batching, and KV cache reuse make the workload viable at 30-million-prediction daily scale.

Models used: Qwen2VL 7B · LLaVA 1.5 7B · LLaMA 3.2 11B · Vision Language Models · Multi-LLM annotation system

Scale

• The system processes over 30 million predictions daily.
  “The system processes over 30 million predictions daily, classifying products into a taxonomy of more than 10,000 categories while extracting over 1,000 product attributes.”summary, stated
• The taxonomy spans more than 26 business verticals.
  “The Shopify Product Taxonomy provides a structured framework spanning more than 26 business verticals with over 10,000 product categories and 1,000+ associated attributes.”summary, stated
• Predicted categories reach an 85% merchant acceptance rate.
  “For merchants, there is an 85% acceptance rate of predicted categories, indicating high trust in system accuracy.”summary, stated

Alternatives considered (roads not taken)

Traditional TF-IDF and logistic regression classifiers

Rejected because: They worked for simple cases but struggled with Shopify's product diversity.

Category-only classification

Rejected because: Shopify needed granular product understanding, attributes, descriptions, content tags, and trust-and-safety features.

Fixed batch serving

Rejected because: In-flight batching better matched real-time arrival patterns and GPU utilization.

Patterns

model-routing (model operations)

Shopify evaluates and transitions between VLM generations by comparing both quality and compute cost, rather than always selecting the newest model.

latency-optimization (serving)

FP8 quantization, in-flight batching, and KV cache reuse improve inference throughput for the two-stage VLM workload.

data-quality-loop (training data)

A multi-LLM annotation, arbitration, and human-validation system improves product taxonomy training data and edge-case handling.

structured-output-contract (taxonomy)

Predictions are constrained and validated against Shopify's taxonomy categories, attributes, standardized values, and rules.

Tradeoffs

Optimize Qwen2VL 7B instead of chasing larger VLMs

Chose A 7B VLM with FP8 quantization and serving optimizations over A larger or newer model without regard to serving cost.

Rationale: The system needed enough multimodal quality at daily production throughput.

Use two sequential prediction calls

Chose Category prediction followed by attribute prediction using category context over One combined prediction call.

Rationale: Attributes are category-specific, so category context improves structured attribute extraction.

Use LLMs for annotation but keep arbitration and human validation

Chose Multi-LLM annotation plus judge models and strategic manual review over Single-model automatic labels.

Rationale: The taxonomy needs robust training labels for edge cases and novel products.

Gotchas (what bit them)

Category-only product understanding was too shallow.

Root cause: Search, recommendations, tax, content safety, and catalog organization needed attributes, descriptions, and trust signals beyond category labels.

Resolution: Use a two-stage VLM pipeline that predicts categories first and then category-specific attributes.

Throughput requirements forced inference optimization.

Root cause: Large VLMs were being used for more than 30 million daily predictions.

Resolution: Combine FP8 quantization, in-flight batching, KV cache optimization, and Kubernetes GPU serving.

The reported acceptance rate still leaves a large review tail.

Root cause: Even a high percentage error or intervention rate becomes large at Shopify's daily prediction volume.

Resolution: Keep merchant intervention and quality feedback in the operating model instead of treating 85% acceptance as full automation.

Evaluation

Shopify evaluates the system through merchant acceptance, hierarchical precision and recall, model-generation comparisons, taxonomy validation, annotation audits, and downstream product-quality indicators.

Metrics: 85% merchant acceptance rate · Over 30 million daily predictions · Doubled hierarchical precision and recall · 10,000+ categories · 1,000+ associated attributes

Outcomes

• Predicted categories reached an 85% merchant acceptance rate.
• Hierarchical precision and recall doubled compared with an earlier neural-network approach.
• Search relevance, recommendations, tax calculations, catalog organization, and content safety are reported as improved downstream surfaces.

The system runs more than 30 million predictions daily across a taxonomy with more than 10,000 categories and 1,000 attributes.