Building Production AI at Scale with Internal Tooling and Agent-Based Systems

shopify · E-commerce · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Company-wide production AI tooling, experimentation, simulation, and model serving

Shopify needed to make AI useful across engineering, ML experimentation, product optimization, merchant/store simulation, search, catalog, and internal development workflows at massive scale. The challenge was not simply increasing token spend, but building internal systems that turn agents into quality improvements, reproducible experiments, customer simulations, low-latency model serving, and safer production deployment rather than uncontrolled code volume and expensive parallel-agent waste.

Constraints:

• Nearly universal AI-tool adoption creates a much larger code and experiment volume that stresses PR review and CI/CD.
• Token consumption alone is a poor metric; low-quality swarms can waste tokens without producing better work.
• AI-generated code can add more total bugs even if the per-line bug rate is lower, because generated volume is high.
• ML/data experimentation needs reproducibility, collaboration, content-based caching, and production handoff.
• Customer simulation is only useful if calibrated against historical real-world outcomes, not just prompted in a vacuum.
• Some workloads require ultra-low latency, while others require maximum batch throughput and long context.

Architecture

Shopify combines company-wide AI tool adoption with internal systems that shape how the work is produced and reviewed. Developers use multiple AI tools with effectively unlimited tokens, but Shopify steers toward high-quality models and critique loops: one agent writes or changes code, a different model critiques it, and the first agent iterates. Internal PR review uses expensive pro-level models and time-consuming review to prevent generated-code volume from overwhelming deployment. Tangle provides a content-hash-based experimentation platform for reproducible data/ML pipelines, and Tangent layers an auto-research agent on top that runs experiments, analyzes results, modifies code/YAML components, and optimizes a goal. SimGym uses decades of Shopify merchant/customer outcome data to calibrate browser-based multimodal customer-simulation agents. Liquid models serve low-latency search understanding and high-throughput batch jobs after distillation from larger models.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  shopify-employees["Shopify employees"] --> ai-tool-portfolio["AI tool portfolio"]
  shopify-employees --> tangle-experimentation-platform["Tangle experimentation platform"]
  shopify-employees --> tangent-auto-research-agent["Tangent auto-research agent"]
  ai-tool-portfolio --> critique-loop-code-workflow["Critique-loop code workflow"]
  critique-loop-code-workflow --> automated-pr-review-models["Automated PR review models"]
  automated-pr-review-models --> ci-cd-deployment-system["CI/CD deployment system"]
  tangle-experimentation-platform --> tangle-content-addressed-cache["Tangle content-addressed cache"]
  tangle-experimentation-platform --> tangent-auto-research-agent
  tangle-experimentation-platform --> production-experiment-runner["Production experiment runner"]
  tangent-auto-research-agent --> production-experiment-runner
  historical-merchant-outcome-data["Historical merchant outcome data"] --> simgym-customer-simulation["SimGym customer simulation"]
  simgym-customer-simulation --> browser-simulation-farm["Browser simulation farm"]
  simgym-customer-simulation --> multimodal-evaluation-models["Multimodal evaluation models"]
  simgym-customer-simulation --> counterfactual-modeling-system["Counterfactual modeling system"]
  counterfactual-modeling-system --> shopify-pulse["Shopify Pulse"]
  browser-simulation-farm --> infrastructure-optimization-partners["Infrastructure optimization partners"]
  liquid-model-serving-stack["Liquid model serving stack"] --> search-query-understanding-service["Search query understanding service"]
  liquid-model-serving-stack --> batch-catalog-enrichment-pipeline["Batch catalog enrichment pipeline"]
  liquid-model-serving-stack --> infrastructure-optimization-partners

Components

Data flow

Employee AI work begins in tools such as Claude Code, Codex, Copilot, Cursor, and Shopify internal tools. Code-generation agents produce candidate changes, then critique/review agents using larger models inspect PRs before CI/CD and deployment. For ML/data work, a practitioner or agent creates a Tangle pipeline from YAML and CLI-wrapped components. Tangle content hashes inputs/outputs so shared preprocessing is reused across people and experiments; Tangent can run many experiments, change components or code, evaluate a loss/goal, and continue optimizing. SimGym uses Shopify's historical merchant/store changes and downstream sales/add-to-cart/conversion outcomes to calibrate customer agents that run in simulated browser environments with multimodal models. Liquid models are distilled for specific tasks, then deployed either as tiny low-latency search models or larger throughput-oriented batch models.

Why this architecture

Shopify's approach separates raw AI usage from productive AI systems. Unlimited access and tool choice encourage adoption, but critique loops and expensive PR review protect quality. Tangle/Tangent convert experimentation into a shared, reproducible, content-addressed platform so agents and humans can iterate without digital archaeology. SimGym only works because Shopify can calibrate simulated agents against historical real merchant outcomes. Liquid serving addresses the operational extremes that general frontier models do not: tiny low-latency search models and high-throughput long-context batch distillation targets.

Models used: Claude Code · Codex · GitHub Copilot · Cursor · GPT 5.4 Pro · Gemini Deep Think · Opus 4.6 · Liquid 300M model · Liquid 7-8B model · Qwen · GPT-4 · GPT OSS · Multimodal model

Scale

• Shopify's AI tool adoption approaches nearly 100% of company workers.
  “basically the active users of Yeah. >> AI tool as a percentage of all the people in the company”transcript, stated

  “it approaches really 100% by now.”transcript, stated
• Shopify effectively funds unlimited tokens for employees while discouraging lower-quality models.
  “You can use any tool, and we effectively fund unlimited tokens for everybody.”transcript, stated

  “please don't use anything less than Opus 4.6.”transcript, stated
• PR merge growth is described as 30% month-on-month rather than 10%.
  “PR merge growth where we're now at 30% uh, month-on-month rather than 10%.”transcript, stated
• Tangent improved search from 800 QPS to 4200 QPS on the same number of machines.
  “we moved from it's hard to even quote from 800 QPS to 4200 QPS with the same quality”transcript, stated
• One auto-research run performed over 400 experiments across several weeks and found one successful improvement.
  “ended up several weeks run.”transcript, stated

  “over 400 experiments, and only one was successful.”transcript, stated
• SimGym optimization targeted 0.7 correlation with add-to-cart events.
  “internal goal was to get hit 0.7 correlation with the add to cart events”transcript, stated
• SimGym took almost a year of calibration and optimization work.
  “that's why it took so long. It took almost a year of that optimization”transcript, stated
• Shopify has decades of historical merchant/store change data to calibrate customer simulations.
  “Shopify has decades of history of how people made changes and what there is what it resulted in terms of sales.”transcript, stated
• The Liquid search model is around 300 million parameters and runs end-to-end in 30 milliseconds.
  “A tiny model like 300 million parameters in but we run it in 30 milliseconds end-to-end for search”transcript, stated
• Larger Liquid models around 7-8B parameters are used for throughput-oriented batch work.
  “we are talking models in maybe 7-8 billion parameter range.”transcript, stated
• Batch catalog/product work operates over billions of products.
  “we have billions and billions of products.”transcript, stated

Alternatives considered (roads not taken)

Judge AI engineering effectiveness by raw token consumption

Rejected because: Shopify sees token consumption as insufficient and focuses on quality loops, critique, and review-token ratios instead.

Run many independent agents in parallel

Rejected because: Shopify treats isolated parallel agents as an anti-pattern compared with fewer agents that communicate through a critique loop.

Optimize PR review for low latency

Rejected because: Shopify would rather spend more time on large-model PR review than pay the aggregate cost of failed tests, evictions, retests, and deployment delays.

Notebook/script-based ML experimentation and manual production porting

Rejected because: Shopify describes that path as irreproducible digital archaeology; Tangle provides versioned, content-addressed, shareable experiments that ship to production directly.

Airflow-style scheduled production pipelines for experimentation

Rejected because: Airflow is framed as good for repeatedly running known production jobs, but weaker for team experimentation, cloning, sharing, and hyperparameter iteration.

Prompt customer-simulation agents without historical calibration

Rejected because: Without historical outcome data, simulated agents only repeat the prompt and have no predictive link to real merchant/customer behavior.

HTML-only customer simulation

Rejected because: Shopify needed visual/browser multimodal simulation because visual changes such as image size affect sales in ways HTML tags alone miss.

Use transformer models for every low-latency or high-throughput workload

Rejected because: Shopify found Liquid non-transformer models best for small low-latency or long-context tasks, and useful as distillation targets for batch workloads.

Patterns

multi-agent-orchestration (coding-workflow)

Shopify uses critique loops where one agent produces work and another model critiques it, after which the original agent iterates.

eval-driven-development (review)

PR review, auto-research goals, SimGym correlation targets, and deployment test feedback all shape what gets accepted or optimized.

semantic-caching (experiment-platform)

Tangle uses content hashes so unchanged outputs and shared preprocessing are reused even when versions or upstream owners differ.

batch-offline-processing (experimentation)

Tangle/Tangent support large-scale offline data/ML experiments, while Liquid batch models handle product categorization, attributes, clustering, and long-context jobs.

synthetic-data-generation (simulation)

SimGym creates calibrated customer-simulation agents from historical merchant/customer outcome data to evaluate storefront changes and recommendations.

model-distillation (model-serving)

Shopify distills large frontier models into Liquid models for specific catalog, Pulse, and batch tasks where throughput, latency, or long context matter.

latency-optimization (model-serving)

Shopify uses and tunes Liquid models for strict latency paths such as 30 ms search query understanding.

model-routing (model-selection)

Shopify remains model-omnivorous, choosing between Liquid, Qwen, frontier models, and other systems based on merit for each workload profile.

Tradeoffs

Coding-agent quality loop

Chose Fewer communicating agents with critique loops over Many independent agents running in parallel.

Rationale: Critique loops increase latency but produce higher-quality code and use tokens more efficiently.

PR review budget allocation

Chose Spend expensive model tokens on PR review over Optimize primarily for cheap code generation tokens.

Rationale: Review prevents expensive downstream test failures and rollbacks, so Shopify measures generation-token spend relative to review-token spend.

Experiment platform design

Chose Tangle as a shared platform with content hashes and one-click production over Individual notebooks/scripts or Airflow-only scheduled jobs.

Rationale: Tangle supports team experimentation, sharing, cloning, reproducibility, cache reuse, and development-to-production handoff.

Auto-research economics

Chose Let Tangent run hundreds of cheap experiments over Human researchers manually attempting every candidate improvement.

Rationale: The success rate can be low, but machine-run experimentation is economical when the alternative would take human years.

SimGym product direction

Chose Single-theme analysis with recommendations over Only predicting winner between merchant-provided A/B variants.

Rationale: Most merchants do not have A and B; they need suggestions for what to change.

Customer simulation fidelity

Chose Visual browser-based multimodal simulation over HTML/text-only agent evaluation.

Rationale: Customer behavior depends on visual factors such as image size, so the simulation must see what humans see.

Liquid model serving profile

Chose Tiny Liquid models for low latency and larger distilled Liquid models for throughput batch jobs over A single frontier-transformer serving strategy.

Rationale: Search requires strict 30 ms latency, while catalog enrichment is an overnight throughput problem over billions of products.

Gotchas (what bit them)

Raw token consumption can hide wasteful AI work.

Root cause: Many isolated parallel agents can burn tokens without communicating or improving output quality.

Resolution: Track how tokens are spent and prefer structured critique loops with fewer high-quality agents/models.

Generated-code volume can increase total production bugs.

Root cause: Even if good models write fewer bugs per line than humans, they write so much more code that more bugs can enter production.

Resolution: Use a strong PR-review narrow waist with expensive models and rigorous automated/human review.

Shortening PR review latency can lengthen total deployment time.

Root cause: Failed tests require finding offending PRs, evicting them, retesting, and delaying deployment.

Resolution: Accept longer large-model review when it reduces aggregate CI/CD churn.

Notebook/script experimentation causes digital archaeology.

Root cause: Ad hoc notebooks, TSVs, scripts, feature-name mismatches, and forgotten paths make experiments hard to reproduce months later.

Resolution: Use Tangle's versioned, content-addressed, reusable pipeline platform and one-click production handoff.

Auto-research is not a substitute for deep creative innovation.

Root cause: Tangent is strong at obvious or bandwidth-limited optimizations, but struggles with out-of-distribution ideas requiring days of non-obvious thinking.

Resolution: Use it as a tireless optimizer for measurable surfaces, while keeping humans responsible for deep conceptual leaps.

Customer simulation without historical calibration is prompt echo.

Root cause: Prompted agents in a vacuum do what the prompt says rather than predict real customer behavior.

Resolution: Calibrate agents with decades of historical store changes and outcome data, plus denoising and collaborative-filtering-style methods.

SimGym violates ordinary LLM serving assumptions.

Root cause: It needs expensive multimodal models, visual browser environments, long contexts, and throughput/latency profiles far from standard text inference.

Resolution: Optimize infrastructure with partners, browser farms, MIG workarounds, distillation, and cheaper headless/headful browser execution.

Low-latency small-model serving exposes CUDA/stack details that large-model serving hides.

Root cause: Optimizations designed for large transformers do not automatically fit tiny 300M low-latency Liquid models.

Resolution: Work with NVIDIA, CentML, and Liquid AI to tune the serving profile and optimize small-model bottlenecks.

Evaluation

Shopify evaluates the stack through adoption metrics, token-distribution metrics, PR/deployment outcomes, Tangle cache/reproducibility behavior, Tangent measurable objective improvements, SimGym correlation against add-to-cart/conversion outcomes, production usage growth, and Liquid latency/throughput measurements. The review emphasizes aggregate system effects rather than a single benchmark.

Metrics: Daily active AI-tool users as a percentage of company employees. · Token consumption distribution by percentile and tool type. · PR merge growth and estimated change complexity. · Ratio of code-generation token spend to PR-review token spend. · Deployment delay from failed tests, offending PR eviction, and retesting. · Tangle cache hits/content-hash reuse and reproducibility. · Tangent objective/loss improvements, QPS, latency, storage reduction, and successful experiments. · SimGym correlation with add-to-cart/conversion and daily usage growth. · Liquid end-to-end search latency and batch throughput.

Outcomes

• AI tool adoption approached nearly 100% of Shopify workers.
• Shopify reports 30% month-over-month PR merge growth while code complexity rises.
• Critique loops produce higher code quality at the cost of higher latency.
• Tangle avoids repeated preprocessing through content hashes and shared experiment reuse.
• Tangent improved search throughput from 800 QPS to 4200 QPS with the same quality on the same machines.
• Tangent found at least one improvement after a 400+ experiment, several-week run on a well-optimized problem.
• SimGym was optimized toward 0.7 correlation with add-to-cart events and sees growing daily usage.
• Liquid models deliver 30 ms end-to-end search query understanding with a 300M model and large-scale batch work with 7-8B distilled models.

The transcript cites nearly 100% internal AI tool adoption, unlimited token funding, 30% month-over-month PR merge growth, 800 to 4200 QPS search throughput, 400+ experiments over several weeks, 0.7 add-to-cart correlation target, decades of merchant outcome data, daily growth in SimGym usage, 30 ms search latency with a 300M model, 7-8B batch Liquid models, and billions of products.