Fine-Tuning Qwen3-32B for Automated Workflow Generation from Natural Language

shopify · E-commerce · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: commerce workflow automation

Shopify needed Sidekick to generate Shopify Flow automations from plain-English merchant requests, but the team lacked initial production conversations, Flow's JSON DSL was out-of-distribution for LLMs, and offline benchmark parity masked a large production activation gap.

Constraints:

the feature had no production conversation data at launch
Flow workflows use a deeply nested JSON DSL designed for backend parsing rather than model generation
training data had to match production tool names, tool order, tool responses, system prompt, and tool descriptions exactly
tool responses could bloat context and degrade reasoning, latency, and cost
offline benchmarks did not cover real merchant requests such as editing workflows, email configurations, third-party integrations, or Flow questions

Architecture

Shopify built a fine-tuned Qwen3-32B tool-calling agent for Sidekick. To cold-start training, they sampled validated production Flow workflows, generated plausible user requests with a stronger LLM, and constructed ideal multi-turn tool trajectories. They translated Flow's JSON DSL into Python for model training, maintained a round-trip transpiler back to Flow JSON for production, mirrored production tool interfaces in training data, minimized tool context through summary/detail calls, trained weekly on H200 infrastructure through Tangle, and used an LLM judge plus tagged slice analysis to route production conversations back into the training pool.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  sidekick-merchant-interface[Sidekick merchant interface]
  validated-production-workflows[Validated production workflows]
  synthetic-query-generator[Synthetic query generator]
  tool-trajectory-constructor[Tool trajectory constructor]
  qwen3-32b-flow-agent[Qwen3-32B Flow agent]
  python-and-flow-json-transpiler[Python and Flow JSON transpiler]
  flow-tool-apis[Flow tool APIs]
  tangle-training-pipeline[Tangle training pipeline]
  llm-judge-and-tagging-system[LLM judge and tagging system]
  production-conversation-feedback[Production conversation feedback]
  flow-backend[Flow backend]
  validated-production-workflows --> synthetic-query-generator
  validated-production-workflows --> tool-trajectory-constructor
  synthetic-query-generator --> tool-trajectory-constructor
  tool-trajectory-constructor --> qwen3-32b-flow-agent
  sidekick-merchant-interface --> qwen3-32b-flow-agent
  qwen3-32b-flow-agent --> python-and-flow-json-transpiler
  qwen3-32b-flow-agent --> flow-tool-apis
  python-and-flow-json-transpiler --> flow-backend
  production-conversation-feedback --> llm-judge-and-tagging-system
  llm-judge-and-tagging-system --> tangle-training-pipeline
  tangle-training-pipeline --> qwen3-32b-flow-agent
  tangle-training-pipeline --> llm-judge-and-tagging-system

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Sidekick merchant interface`	Accepts merchant natural-language requests for Shopify Flow automations.	`Qwen3-32B Flow agent`
`Validated production workflows`	Existing anonymized manually built Flow workflows used to bootstrap synthetic training examples.	`Synthetic query generator`, `Tool trajectory constructor`
`Synthetic query generator`	Uses a stronger LLM to produce plausible natural-language requests from existing workflows.	`Tool trajectory constructor`
`Tool trajectory constructor`	Builds ideal multi-turn tool-call sequences for the agent to learn from.	`Qwen3-32B Flow agent`
`Qwen3-32B Flow agent`	Fine-tuned tool-calling model that generates Python workflow programs from merchant requests.	`Python and Flow JSON transpiler`, `Flow tool APIs`, `CentML serving`
`Python and Flow JSON transpiler`	Converts Flow JSON to Python for training and converts generated Python back to Flow JSON for the backend.	`Flow backend`, `Qwen3-32B Flow agent`
`Flow tool APIs`	Expose task search and configuration retrieval tools with production-mirrored names, ordering, and response formats.	`Qwen3-32B Flow agent`
`Tangle training pipeline`	Runs reproducible data collection, training, evaluation, and deployment workflows with intelligent caching.	`CometML tracking`, `HuggingFace datasets and checkpoints`, `CentML serving`, `LLM judge and tagging system`, `Qwen3-32B Flow agent`
`LLM judge and tagging system`	Scores conversations by lifecycle facet, tags workflow dimensions, identifies gaps, and routes examples into training or quarantine.	`Production conversation feedback`, `Tangle training pipeline`
`Production conversation feedback`	Provides activated workflow conversations and real merchant behavior signals for continuous retraining.	`LLM judge and tagging system`
`Flow backend`	Executes the production JSON workflow representation without needing to understand the Python intermediate form.	`Python and Flow JSON transpiler`

Data flow

Existing production Flow workflows are sampled and filtered, then a stronger LLM generates natural-language merchant requests and engineers construct ideal tool-call trajectories. A transpiler converts Flow JSON into Python so Qwen3-32B learns an in-distribution representation. At inference, the model writes Python, the transpiler converts it back to JSON for the Flow backend, and tools return lightweight summaries before full configuration retrieval. Production conversations are scored by a calibrated LLM judge, tagged by workflow dimensions, routed into training or quarantine, and retrained weekly through Tangle, CometML, Hugging Face, and CentML.

“We reverse-engineered user intent from existing production workflows. Thousands of anonymized store owners had already built workflows manually in Flow.”article, stated

“At inference time, the model writes Python. The transpiler converts it to JSON for the Flow backend.”article, stated

Why this architecture

Shopify chose a fine-tuned tool-calling model because proprietary Flow workflow data and production feedback could create durable differentiation over closed-model APIs. The Python intermediate representation moved the task closer to model pretraining, production mirroring avoided token-level drift, and the weekly flywheel turned live merchant behavior into continuous training signal.

“Lasting differentiation comes from proprietary data, the training recipe, the infrastructure, and the speed of iteration.”article, stated

“Moving the target format from out-of-distribution to in-distribution turned the problem from "learn a new language and the task" into "learn the task."”article, stated

Models used: Qwen3-32B · stronger LLM for synthetic query generation · LLM judge · frontier closed model baseline

Scale

Shopify has millions of merchant interactions that can signal output quality.
“Shopify has something most companies don't: a product surface where millions of merchant interactions directly signal whether the model's output is any good.”article, stated
The fine-tuned agent is reported as 2.2x faster and 68% cheaper than the closed-model baseline.
“It's 2.2x faster, 68% cheaper, and outperforms closed models.”article, stated
The initial benchmark used 300 hand-crafted examples.
“evaluated it against a benchmark of 300 hand-crafted examples covering the breadth of expected Flow usage.”article, stated
Switching from JSON DSL to Python improved syntactic correctness by 22 points and semantic correctness by 13 points.
“switching from the JSON DSL to the Python DSL improved syntactic correctness by 22 points and semantic correctness by 13 points.”article, stated
Qwen3-32B trains on two nodes of H200 GPUs, with a full run taking 12 hours.
“Qwen3-32B trains on two nodes of H200 GPUs with Fully Sharded Data Parallel (FSDP). A full training run takes 12 hours”article, stated
The first 1% traffic deployment showed workflow activation 35% lower than the prompt-based agent.
“At 1% traffic, the fine-tuned model's workflow activation rate (whether store owners actually turn on the workflows Sidekick generates) came in 35% lower than the prompt-based agent.”article, stated
Email workflows accounted for 25% of failures and diverse condition patterns for 16%.

"Email workflows accounted for 25% of failures, so we added email-specific examples

Diverse condition patterns accounted for 16%"

— article, stated

The fine-tuned Flow agent now serves the majority of production traffic.
“The fine-tuned Flow agent now serves the majority of our production traffic.”article, stated

Alternatives considered (roads not taken)

wait for production conversations before training

Rejected because: the feature had not launched, so there were no conversations; Shopify instead worked backward from existing production workflows.

“Fine-tuning required training data, but since the feature hadn't been deployed yet, there were no production conversations to learn from.”article, stated

train directly on Flow's JSON DSL

Rejected because: the deeply nested JSON DSL was a poor fit for LLM pretraining and hurt syntactic and semantic correctness.

“That format is ideal for production systems, but it's a poor fit for LLMs.”article, stated

return full tool schemas immediately

Rejected because: full upfront tool responses inflated context, latency, and cost while diluting reasoning signal.

“Instead of returning full details for every result upfront, tools return lightweight summaries first.”article, stated

trust offline benchmark parity as launch proof

Rejected because: production traffic revealed a 35% activation-rate gap caused by out-of-distribution merchant requests.

“Synthetic data got us to parity on offline benchmarks. By every metric we tracked, the fine-tuned model was ready for production. We deployed it to 1% of traffic to see how it held up.”article, stated

optimize directly for activation rate only

Rejected because: activation reflected merchant behavior as well as model quality, so Shopify used a calibrated LLM judge while keeping activation as a guardrail.

“Activation rate was our first production signal, but it turned out to be noisy: it reflects merchant behavior, not model quality.”article, stated

Patterns

`tool-use-orchestration` (agent-architecture)

The Flow agent generates multi-turn tool trajectories, searches Flow components, retrieves configurations only when needed, and incorporates tool responses into workflow generation.

“The task requires tool calling. The model must reason, act, and incorporate external results, not just generate text.”article, stated

`fine-tuning-adaptation` (model-operations)

Shopify fine-tuned Qwen3-32B on synthetic and later production-derived Flow trajectories so the model could generate workflow automations from natural language.

“We fine-tuned Qwen3-32B on this synthetic dataset and evaluated it against a benchmark of 300 hand-crafted examples”article, stated

`structured-output-contract` (output-control)

The system uses Python as the model-facing representation and a round-trip transpiler to enforce semantically equivalent production Flow JSON.

“Making this work required building a round-trip transpiler between Python and Flow's JSON representation to handle the full complexity of Flow logic”article, stated

`eval-driven-development` (evaluation)

Shopify used semantic judges, syntactic program checks, production activation guardrails, human-calibrated LLM judges, and tagged slice analysis to guide retraining.

“The judge grades each facet separately rather than treating quality as a single pass/fail outcome.”article, stated

`latency-optimization` (serving)

Tool interfaces return lightweight summaries first and full configurations only for selected tools, reducing context growth, latency, and cost.

“The model picks the 2-3 it actually needs, then calls task_configuration to get the full schema only for those. The context stays small, the reasoning stays focused.”article, stated

`cost-aware-routing` (serving)

Shopify replaced a frontier model they did not control with an owned fine-tuned model that was cheaper and faster in production.

“Now it runs on a model we trained, on infrastructure we own, improving from data only we have, at 68% lower cost.”article, stated

Tradeoffs

Bootstrap from existing workflows rather than wait for new conversations.

Chose reverse-engineered synthetic trajectories from validated workflows over waiting for production Sidekick Flow conversations.

Rationale: The feature had not launched, but existing manually created workflows already encoded merchant intent and valid outputs.

“If you're building an agent without interaction data, start with the output artifacts your users already produce and work backwards from them.”article, stated

Train the model to write Python instead of Flow JSON.

Chose in-distribution Python intermediate representation over direct generation of the production JSON DSL.

Rationale: Python is closer to model pretraining and improved both syntactic and semantic correctness while the transpiler preserved production semantics.

“Python is far closer to code and logical reasoning, and it makes up a large share of pretraining data.”article, stated

Use two-step tool retrieval for Flow components.

Chose summary search followed by selected configuration retrieval over returning full schemas for all search matches.

Rationale: The split keeps context small and focused even when hundreds of triggers, conditions, and actions are available.

“A search might return 100 matches. Rather than loading the full configuration schema for each one, task_search returns just names and descriptions.”article, stated

Optimize with a calibrated LLM judge instead of activation rate alone.

Chose domain-expert-calibrated judge with activation as a guardrail over raw workflow activation rate as the primary quality signal.

Rationale: Activation rate reflects merchant behavior as well as model quality, so it is useful but noisy.

“We therefore optimized for a domain-expert-calibrated LLM judge, which we describe next, while keeping activation rate as a guardrail”article, stated

Own the model and infrastructure.

Chose fine-tuned in-house Qwen3-32B served on Shopify's stack over continuing on a closed frontier model.

Rationale: Owned data, training, infrastructure, and iteration speed produced lower cost and an improving feedback loop.

“Six months ago, this system ran on a frontier model we didn't control. Now it runs on a model we trained, on infrastructure we own”article, stated

Gotchas (what bit them)

Offline benchmark parity hid a large production gap.

Root cause: The synthetic benchmark covered expected Flow requests but missed real merchant behavior such as editing workflows, email configurations, third-party integrations, and Flow questions.

Resolution: Shopify deployed to 1% traffic, used activation as an early signal, then built a judge/tagging diagnostic layer to add targeted data and retrain.

“The benchmark covered what we expected merchants to ask. It didn't cover what they actually asked: editing existing workflows, handling email configurations, working with third-party integrations”article, stated

Tiny training/serving mismatches hurt model behavior.

Root cause: The model treated tool names, tool order, response key order, extra fields, system prompts, and descriptions as meaningful tokens.

Resolution: Shopify mirrored production tooling in training data and treated prompt/tool-description sync as an ongoing process.

“The model treats every token as a signal, whether you intended it or not.”article, stated

Training directly on Flow's JSON DSL made the model learn both an unfamiliar format and the task.

Root cause: Conditional program-like logic was embedded in deeply nested JSON, a pattern rare in pretraining.

Resolution: Shopify translated workflows into Python for model generation and used a transpiler back to JSON.

“Conditional, program-like logic that would normally appear as code is embedded in deeply nested JSON, a pattern that's rare in pretraining data.”article, stated

Tool-call context growth hurt reasoning, latency, and cost.

Root cause: Every tool response becomes part of context, and irrelevant context dilutes the signal the model uses.

Resolution: The tool stack was restructured to return summaries first and full schemas only for selected components.

“When an agent calls tools, every response becomes part of the context. Context grows, latency grows, cost grows. Worse, irrelevant context dilutes the signal.”article, stated

More data slowed iteration.

Root cause: Larger training sets improved accuracy but made each run slower, reducing weekly experiment velocity.

Resolution: Shopify trained on two H200 nodes with FSDP and used Tangle caching to keep full training runs to about 12 hours.

“more training data improved accuracy but slowed each run. Slower runs meant fewer iterations, and fewer iterations meant slower improvement.”article, stated

Evaluation

Shopify used a layered evaluation approach: a 300-example offline benchmark with LLM-judged semantic correctness, programmatic syntactic correctness, and latency; a 1% production deployment with activation-rate guardrail; a human-calibrated LLM judge that scores lifecycle facets; and tagged slice analysis to identify coverage gaps.

Metrics: semantic correctness judged by an LLM · syntactic correctness checked programmatically · latency from request to workflow delivery · 2.2x faster than the closed-model baseline · 68% cheaper than the closed-model baseline · 35% lower activation rate at initial 1% traffic · email workflows accounted for 25% of failures · diverse condition patterns accounted for 16% of failures

"We looked at three metrics:

Semantic correctness: Does the generated workflow do what it's supposed to?"

“We built an LLM-based judge that scores each conversation across the workflow lifecycle: whether the assistant correctly understood the merchant's intent”article, stated

Outcomes

The fine-tuned tool-calling agent is reported as 2.2x faster and 68% cheaper than the closed-model baseline.
Python DSL training improved syntactic correctness by 22 points and semantic correctness by 13 points.
A 1% production rollout exposed a 35% activation-rate gap versus the prompt-based agent.
The flywheel closed the production gap in two weeks.
The fine-tuned Flow agent now serves the majority of production traffic.
The pattern is already being applied to other Sidekick skills.

The system operates on Shopify's merchant-scale production surface, trains Qwen3-32B weekly on two H200 nodes, and uses every production conversation as potential training signal through an LLM-judge routing loop.

“And when benchmarks said we were ready but production said otherwise, the flywheel closed the gap in two weeks.”article, stated

“Within Sidekick, this pattern is already being applied to other skills.”article, stated