Building a Global Product Catalogue with Multimodal LLMs at Scale

shopify · E-commerce · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Global ecommerce product-catalogue understanding

Shopify needed to turn fragmented, merchant-authored product listings from millions of merchants into unified, standardized, machine-readable product knowledge that can power search, recommendations, personalization, conversational commerce, and product matching.

Constraints:

• merchants describe billions of products in unique and inconsistent ways
• the system must process high-volume product updates continuously
• commercial multimodal APIs are too expensive at Shopify's inference scale
• product understanding spans multiple entities and tasks with interdependent outputs
• production serving must balance accuracy, latency, GPU usage, and token volume

Architecture

Shopify built Global Catalogue as a four-layer system: product data foundation, product understanding, product matching, and reconciliation. The product-understanding layer fine-tunes open-source vision-language models over multi-task, multi-entity catalogue entities. The matching layer clusters equivalent listings, and reconciliation builds canonical product records. Production inference runs through Triton, FP8 quantization, batching/routing, and selective field prompting so surfaces ask only for the fields they need.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  merchant-product-updates[Merchant product updates] --> product-data-foundation[Product data foundation]
  product-data-foundation --> product-understanding-layer[Product understanding layer]
  annotation-pipeline[Annotation pipeline] --> fine-tuned-vlms[Fine-tuned VLMs]
  fine-tuned-vlms --> product-understanding-layer
  product-understanding-layer --> product-matching-layer[Product matching layer]
  product-matching-layer --> reconciliation-layer[Reconciliation layer]
  reconciliation-layer --> global-catalogue[Global Catalogue]
  fine-tuned-vlms --> triton-inference-server[Triton inference server]
  triton-inference-server --> commerce-surfaces[Commerce surfaces]
  global-catalogue --> commerce-surfaces

Components

Data flow

Product updates arrive from merchant uploads, APIs, apps, and integrations into the product data foundation. Product-understanding models infer standardized metadata from text and images. Matching builds clusters of equivalent products across merchants, and reconciliation aggregates inferred metadata into canonical product records. Downstream surfaces such as search, recommendations, APIs, bulk pipelines, admin UI, Sidekick, and Shop app chat consume the enriched catalogue.

Why this architecture

Shopify fine-tuned smaller open-source multimodal models because commercial APIs were too costly at its product-update volume. The multi-task, multi-entity setup captured interdependencies among catalogue fields, while selective field prompting restored generalization and reduced token generation. The four-layer catalogue separated ingestion, understanding, matching, and canonical record construction.

Models used: LlaVA 1.5 7B · LLaMA 3.2 11B · Qwen2VL 7B · OpenAI · Gemini · LLM agents

Scale

• Shopify hosts millions of merchants and billions of products.
  “Shopify hosts millions of merchants who describe billions of products in wonderfully unique ways. That diversity, while an incredible strength for commerce, is kryptonite for machine understanding.”article, stated
• The catalogue processes over 10 million product updates daily.
  “We process over 10 million product updates daily from merchant uploads, APIs, apps, and integrations, in a streaming fashion.”article, stated
• Production inference handles 40 million LLM calls daily.
  “Our inference infrastructure handles supports 40 million LLM calls daily, representing about 16 billion tokens inferred per day”article, stated
• The infrastructure infers about 16 billion tokens per day.
  “Our inference infrastructure handles supports 40 million LLM calls daily, representing about 16 billion tokens inferred per day, through several optimization techniques:”article, stated
• Selective field extraction reduced median latency from 2 seconds to 500 milliseconds.
  “models trained with selective field extraction retain better generalization capabilities, but we reduced median latency from 2 seconds to 500 milliseconds. Moreover, by generating fewer tokens, we substantially reduced GPU usage by 40%, allowing us to serve more requests with the same hardware,”article, stated
• Selective prompting reduced GPU usage by 40 percent.
  “Moreover, by generating fewer tokens, we substantially reduced GPU usage by 40%, allowing us to serve more requests with the same hardware, which has made our system more cost-effective.”article, stated

Alternatives considered (roads not taken)

Commercial multimodal APIs at full scale

Rejected because: Shopify's inference volume made OpenAI and Gemini prohibitively expensive.

Separate model per extraction task

Rejected because: Multi-task fine-tuning exploited interdependencies among category, summarization, and classification decisions.

Ask the model to predict all fields at inference

Rejected because: Predicting all fields harmed generalizability; selective extraction improved latency and cost.

Patterns

fine-tuning-adaptation (modeling)

Shopify fine-tuned open-source vision-language models for product understanding across catalogue entities and tasks.

cost-aware-routing (operations)

Selective field prompting and open-source model serving reduced token generation and GPU usage at high inference volume.

latency-optimization (operations)

Triton serving, batching/routing, FP8 quantization, and selective field prompting kept daily multimodal inference practical.

synthetic-data-generation (data)

The annotation pipeline used multiple LLM agents as annotators before human expertise and evaluation shaped training data.

data-quality-loop (data)

The catalogue uses change data capture, schema evolution, annotation/evaluation, matching, and reconciliation to maintain canonical product records.

Tradeoffs

Fine-tune open-source VLMs

Chose smaller controlled open-source models over commercial multimodal APIs.

Rationale: Fine-tuning provided better cost control and performance at Shopify's inference volume.

Use multi-task, multi-entity modeling

Chose one model setup that performs related catalogue tasks together over siloed task-specific models.

Rationale: Related tasks improve one another when modeled together.

Use selective field prompting

Chose surface-specific field requests over generating every field for every inference.

Rationale: Selective prompting reduced latency and GPU token usage while preserving generalization.

Gotchas (what bit them)

Merchant data diversity is hostile to machine understanding.

Root cause: Millions of merchants describe billions of products in unique and inconsistent ways.

Resolution: Use a unified catalogue layer that standardizes and enriches product data.

Commercial APIs become uneconomic at catalogue scale.

Root cause: The system performs tens of millions of multimodal inferences daily.

Resolution: Fine-tune and serve smaller open-source VLMs with optimized infrastructure.

All-field extraction reduced generalization.

Root cause: Asking models to predict every field during fine-tuning made them less flexible at inference time.

Resolution: Adopt selective field extraction so each surface requests only required fields.

Evaluation

Shopify evaluates models across accuracy, computational cost, instruction following, training/evaluation data quality, latency, GPU footprint, and production serving throughput.

Metrics: 40 million LLM calls daily · 16 billion tokens per day · 2s to 500ms median latency reduction · 40% GPU token usage reduction · instruction-following metrics

Outcomes

• Global Catalogue standardizes and enriches product data across Shopify
• multimodal inference powers search, recommendations, personalization, and conversational commerce
• selective field prompting reduced latency and GPU usage while serving tens of millions of daily calls

The system processes over 10 million product updates daily and serves 40 million LLM calls daily across Shopify surfaces.