Structured Workflow Orchestration for Large-Scale Code Operations with Claude

shopify · E-commerce · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Large-scale codebase operations and developer productivity

Shopify needed to run repeatable code operations such as test generation, coverage optimization, type-safety migration, and code migrations across a massive engineering estate where pure agentic workflows accumulated entropy and pure deterministic scripts lacked adaptability.

Constraints:

• Shopify has millions of lines of code and about 5,000 repositories.
• The main app has over half a million tests.
• Structured workflows are needed for well-defined, repeatable operations.
• Agentic tools like Claude Code are better for exploratory tasks but remain nondeterministic.

Architecture

Shopify uses ROAST as a structured workflow orchestrator alongside Claude Code. ROAST handles predefined steps such as coverage measurement, test execution, Sorbet autocorrect, and replayable workflow state; Claude Code can call ROAST workflows as tools, while ROAST can invoke Claude Code through the SDK inside a workflow step, creating a bidirectional deterministic/nondeterministic hybrid.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  developer_or_claude_code["Developer or Claude Code"] --> roast_workflow_orchestrator["ROAST workflow orchestrator"]
  roast_workflow_orchestrator --> coverage_and_test_tools["Coverage and test tools"]
  roast_workflow_orchestrator --> claude_code_sdk_step["Claude Code SDK step"]
  roast_workflow_orchestrator --> replayable_workflow_state["Replayable workflow state"]
  coverage_and_test_tools --> shopify_codebase["Shopify codebase"]
  claude_code_sdk_step --> shopify_codebase

Components

Data flow

A developer or Claude Code invokes a ROAST workflow for a concrete operation such as test coverage. ROAST runs deterministic steps like coverage tools and tests, then calls Claude Code for bounded code-generation or repair steps. The workflow can replay from intermediate steps during debugging, and Claude Code can also execute predefined ROAST commands as tools.

Why this architecture

Shopify uses a hybrid architecture because deterministic workflows keep repeatable operations grounded, while Claude Code supplies adaptive coding ability within bounded steps. Interleaving the two reduces entropy accumulation in multi-step code operations.

Models used: Claude Code · Claude Code SDK · ROAST

Scale

• Shopify has about 5,000 repositories.
  “It's probably about 5,000 uh repos”transcript, stated
• The codebase has millions upon millions of lines of code.
  “has millions upon millions of lines of code.”transcript, stated
• The main app has over half a million tests.
  “our over half a million tests associated with our main”transcript, stated
• The presenter reports about 500 daily active users and a 250,000 requests-per-second peak.
  “have at peak now about 500 daily active users and that number is growing rapidly. And as of lately, we hit 250,000 requests per second at at peak,”transcript, stated
• The example test workflow raises branch coverage above 90%.
  “function to raise the branch coverage level of the following test above 90%.”transcript, stated

Alternatives considered (roads not taken)

Use Claude Code alone for repeatable workflow operations

Rejected because: Agentic tools are nondeterministic and can accumulate entropy across multi-step workflows.

Use only deterministic scripts

Rejected because: Some code operations benefit from agentic coding steps such as generating tests or resolving leftover type errors.

Rerun whole workflows during development

Rejected because: ROAST replay avoids rerunning the first four steps when debugging the fifth step of a workflow.

Patterns

tool-use-orchestration (developer workflow)

ROAST orchestrates coverage tools, tests, commands, replay, and Claude Code within repeatable workflow steps.

durable-execution (workflow reliability)

ROAST supports replaying from later workflow steps so developers can debug without rerunning everything.

eval-driven-development (code quality)

The workflows use coverage tools, tests, and explicit coverage targets to guide Claude Code edits.

structured-output-contract (workflow structure)

ROAST turns coding operations into predefined workflow steps rather than free-form agent sessions.

Tradeoffs

Interleave deterministic and nondeterministic work.

Chose ROAST plus Claude Code over Pure Claude Code or pure scripts.

Rationale: Structured workflows keep repeatable operations reliable while Claude handles adaptive code-generation or repair subtasks.

Make ROAST callable by Claude Code and vice versa.

Chose Bidirectional integration over A one-way wrapper around the coding agent.

Rationale: Claude Code can execute predefined workflows, while ROAST workflows can invoke Claude Code for agentic steps.

Gotchas (what bit them)

Nondeterministic agents accumulate errors across multi-step workflows.

Root cause: Each free-form step can drift slightly, making later steps harder to complete.

Resolution: Break large tasks into structured workflows and give the agent only bounded work at each step.

Workflow debugging becomes expensive without step replay.

Root cause: A five-step workflow would otherwise rerun earlier steps when only the fifth is under development.

Resolution: Use ROAST's replay capability to resume from an intermediate step.

Evaluation

The talk evaluates ROAST through adoption, workflow examples, test coverage workflows, and operational fit at Shopify scale.

Metrics: About 5,000 repositories. · Over half a million tests in the main app. · About 500 daily active users. · Peak of 250,000 requests per second. · Example coverage target above 90% branch coverage.

Outcomes

• ROAST supports structured coding workflows for test generation, coverage optimization, Sorbet typing, and migrations.
• Claude Code and ROAST can call each other as tools.
• The workflow pattern supports hundreds of daily users at Shopify scale.

The talk reports about 5,000 repos, millions of lines of code, over half a million tests, 500 daily active users, and a 250,000 RPS peak.