Structured Data Extraction from E-commerce Storefronts Using Specialized Agentic Architecture

shopify · E-commerce · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce storefront understanding and structured extraction

Shopify needed to extract structured shop-level information from millions of highly customized merchant storefronts, including fraud signals, merchant profiles, and tax-related category codes, while reducing cost enough to cover all shops rather than a small subset.

Constraints:

• Merchant storefronts are intentionally flexible and non-standardized.
• A single-shot GPT approach had cost and quality ceilings.
• Millions of shops and changing pages made repeated expensive frontier calls impractical.
• Different extraction objectives needed different tools and optimization paths.

Architecture

Shopify evolved from a single-shot GPT-5 approach to a DSPy-built agentic architecture, then to three specialized ReAct sub-agents for fraud, merchant profile, and merchant category code extraction. A Flink batch job processes shops, calls a Kubernetes-hosted LLM API agent layer, and the agents call a self-hosted Qwen-3-9B model on a separate GPU cluster, enabling cheaper frequent coverage across all shops.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  flink_batch_job["Flink batch job"] --> kubernetes_llm_api["Kubernetes LLM API"]
  kubernetes_llm_api --> fraud_sub_agent["Fraud sub-agent"]
  kubernetes_llm_api --> profile_sub_agent["Profile sub-agent"]
  kubernetes_llm_api --> mcc_sub_agent["MCC sub-agent"]
  fraud_sub_agent --> qwen_gpu_service["Qwen GPU service"]
  profile_sub_agent --> qwen_gpu_service
  mcc_sub_agent --> qwen_gpu_service
  store_snapshots["Store snapshots"] --> kubernetes_llm_api

Components

Data flow

A Flink batch job selects shops and calls the Kubernetes LLM API. The agent layer crawls pages, strips HTML, and runs three specialized ReAct sub-agents for fraud, profile, and MCC extraction. Those agents call the self-hosted Qwen model on a separate GPU cluster. Outputs are produced from store snapshots and replayed store runs for quality optimization.

Why this architecture

Shopify split the system into specialized agents and separate compute layers because monolithic GPT calls were expensive, less controllable, and hit quality ceilings. Specialized sub-agents could use task-specific tools independently, while self-hosted Qwen and separated GPU infrastructure reduced cost enough for all-shop coverage.

Models used: GPT-5 · GPT-4 · Qwen-3-9B · DSPy ReAct agents

Scale

• The solution targets millions and millions of Shopify merchants.
  “like we have millions and millions of merchants, do we really want to make millions and millions of calls”transcript, stated
• The previous cost estimate covered only 13% of shops at about $5 million.
  “for 13% of our shops, we were paying about half or like 5 million 5 million”transcript, stated
• Moving from GPT-5 to Qwen made inference about 75x cheaper.
  “because we moved from GPT-5 to Qwen, now it's much much cheaper, like 75 times cheaper”transcript, stated
• Quality improved close to 2x while lowering cost.
  “our quality goes up like close to 2x the quality at orders of magnitude lower the cost.”transcript, stated
• A later architecture note says the Flink batch job takes about 150K shops daily.
  “we have a Flink batch job that takes about 150K shops daily.”transcript, stated

Alternatives considered (roads not taken)

Single-shot GPT-5 extraction

Rejected because: It was too expensive for millions of merchants and could not cover frequently changing stores.

One monolithic agent

Rejected because: The team found monolithic agents did not perform as well as specialized agents and created cross-quality coupling.

Run all agent work on GPU hardware

Rejected because: Crawling, HTML stripping, and orchestration do not need GPUs, so Shopify split agent infrastructure from model inference.

Patterns

multi-agent-orchestration (agent architecture)

Shopify split storefront extraction into fraud, profile, and MCC ReAct sub-agents with separate tools and optimization paths.

model-routing (model operations)

The architecture routes generation calls from the Kubernetes agent layer to a separate self-hosted Qwen GPU service.

cost-aware-routing (cost optimization)

Shopify moved from GPT-5 to self-hosted Qwen and split CPU/GPU work to reduce cost by about 75x.

batch-offline-processing (processing architecture)

The storefront extraction system runs as a Flink batch job over shop snapshots rather than a synchronous per-user request flow.

eval-driven-development (optimization)

The team used replayed stores and human labels to optimize and compare sub-agent quality.

Tradeoffs

Self-host Qwen for extraction.

Chose Qwen-3-9B on Shopify GPU infrastructure over GPT-5 API calls.

Rationale: The self-hosted model reduced cost enough for all-shop coverage while improving quality in the agentic architecture.

Separate agent layer from GPU inference.

Chose Kubernetes agent layer plus separate GPU cluster over Run the full pipeline on GPU infrastructure.

Rationale: Crawling, HTML stripping, and orchestration are CPU/platform tasks, while only model generation needs GPU inference.

Use specialized sub-agents.

Chose Fraud, profile, and MCC agents over Single monolithic agent.

Rationale: Specialization allowed task-specific tools, independent optimization, and better quality than monolithic extraction.

Gotchas (what bit them)

Single-shot frontier calls do not scale across changing storefronts.

Root cause: Millions of merchants and page changes make repeated GPT-5 calls too expensive.

Resolution: Use a batch pipeline, self-hosted Qwen, and specialized agents to cover all shops more frequently.

Monolithic agents couple unrelated quality changes.

Root cause: One agent handling fraud, profile, and tax objectives cannot isolate tools or optimization changes by task.

Resolution: Use separate ReAct sub-agents so each can be optimized and tooled independently.

Evaluation

Shopify compared single-shot, agentic, sub-agent, and optimized architectures against quality and cost metrics, using replayed stores and human labels for agent optimization.

Metrics: Approximate cost reduction: 75x after moving from GPT-5 to Qwen. · Quality improvement: close to 2x. · Prior subset coverage: 13% of shops at about $5 million. · Batch volume: about 150K shops daily.

Outcomes

• Shopify moved from subset coverage to all-shop coverage.
• The architecture improved quality close to 2x while cutting cost about 75x.
• The final system runs a Flink-to-Kubernetes-to-Qwen batch architecture with three specialized sub-agents.

The source describes millions of merchants, about 150K shops processed daily, and prior costs of about $5 million for only 13% of shops.