Structured AI Workflow Orchestration for Developer Productivity at Scale

shopify · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Developer-productivity workflow automation

Shopify needed a way to make AI agents reliable on large-codebase developer workflows such as flaky tests, low coverage, code quality, and documentation, where free-roaming agents across millions of lines of code were too nondeterministic.

Constraints:

AI agents need structure and discrete steps to stay on track.
The workflows must interleave nondeterministic AI behavior with deterministic code execution.
Developer workflows need to be reproducible, testable, version-controlled, and integrated with normal development processes.
Shopify wanted a convention-oriented tool usable across languages even though it is implemented in Ruby.

Architecture

Shopify built Roast, a command-line workflow orchestration framework that defines AI workflows with workflow.yml and markdown prompts. Roast supports directory steps, command execution, inline prompts, custom Ruby steps, parallel steps, shared conversation transcripts, session replay, and a CodingAgent tool powered by Claude Code, letting teams combine deterministic scripts with bounded AI autonomy.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  workflow_definition["Workflow definition"] --> roast_runner["Roast runner"]
  roast_runner --> deterministic_command_steps["Deterministic command steps"]
  roast_runner --> ai_prompt_steps["AI prompt steps"]
  roast_runner --> codingagent_tool["CodingAgent tool"]
  deterministic_command_steps --> shared_workflow_context["Shared workflow context"]
  ai_prompt_steps --> shared_workflow_context
  codingagent_tool --> shared_workflow_context
  roast_runner --> session_replay_store["Session replay store"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Workflow definition`	workflow.yml plus prompt files that declare ordered, parallel, AI, command, and custom Ruby steps.	`Roast runner`
`Roast runner`	Command-line orchestrator that interprets workflow definitions and executes step types.	`Workflow definition`, `Deterministic command steps`, `AI prompt steps`, `CodingAgent tool`, `Session replay store`
`Deterministic command steps`	Non-AI code execution, scripts, sed, tests, coverage tools, and custom Ruby classes.	`Roast runner`, `Shared workflow context`
`AI prompt steps`	Markdown or inline prompts for analysis, generation, and review within bounded workflow steps.	`Roast runner`, `Shared workflow context`
`CodingAgent tool`	Claude Code integration that gives agentic autonomy within workflow guardrails.	`Roast runner`, `Shared workflow context`
`Shared workflow context`	Conversation transcript and step context reused by later workflow steps.	`Deterministic command steps`, `AI prompt steps`, `CodingAgent tool`
`Session replay store`	Saved execution state that allows resuming and debugging expensive workflows from intermediate steps.	`Roast runner`

Data flow

A developer defines a workflow in YAML and prompt files. Roast interprets each step, runs deterministic commands or Ruby classes where appropriate, calls model prompts or Claude Code through CodingAgent for bounded AI work, shares transcripts across steps, and stores session executions so workflows can resume or replay from a chosen step.

4 supporting quotes

“Roast is a convention-oriented workflow orchestration framework designed specifically for creating structured AI workflows”article, stated

“It provides a declarative approach using YAML configuration and markdown prompts”article, stated

“Steps in a workflow share their conversation transcript, building upon each other's context.”article, stated

“Every workflow execution is automatically saved, allowing you to resume from any step”article, stated

Why this architecture

Roast separates workflow structure from AI autonomy because Shopify found that unbounded AI over huge codebases was unreliable, while discrete steps, deterministic commands, shared context, and replay make AI-assisted development workflows reproducible and testable.

3 supporting quotes

“Allowing AI to roam free around millions of lines of code just didn’t work very well. Non-determinism is the enemy of reliability.”article, stated

“turn complex, multi-step agentic AI processes into reproducible, testable workflows.”article, stated

“You define the guardrails and objectives, but the agent has autonomy within those boundaries to iterate, test, and improve”article, stated

Models used: Claude Code · Roast CodingAgent · Raix

Scale

Roast targets codebases with millions of lines of code.
“Allowing AI to roam free around millions of lines of code just didn’t work very well.”article, stated
A dozen engineers had already contributed features and workflows to Roast on GitHub.
“a dozen Engineers have already contributed features and workflows to Roast on GitHub.”article, stated
Roast requires Ruby 3.0 or newer.
“Ruby 3.0+”article, stated

Alternatives considered (roads not taken)

Let AI agents roam freely around the codebase

Rejected because: Shopify found free-roaming AI across millions of lines of code unreliable.

“Allowing AI to roam free around millions of lines of code just didn’t work very well.”article, stated

Pure deterministic automation

Rejected because: Some workflow steps are unknown or hard to automate deterministically, so AI can act as a placeholder until the team understands the problem better.

“I can handwave a step I don't quite know how to do yet with an AI approximation that mostly works.”article, stated

“it's very easy to drop the AI step for a deterministic one that always works.”article, stated

Ad-hoc chat prompts

Rejected because: Shopify argues future developer work should use version-controlled, tested workflows rather than one-off prompt sessions.

“Rather than ad-hoc prompts in chat interfaces, we see a world where:”article, stated

“AI workflows are first-class citizens in the development process, version-controlled and tested like any other code”article, stated

Patterns

`tool-use-orchestration` (workflow orchestration)

Roast coordinates command execution, custom Ruby steps, AI prompts, Claude Code, shared transcripts, and replay in one workflow.

“structured AI workflows that interleave non-deterministic AI behavior with normal non-AI code execution.”article, stated

`structured-output-contract` (workflow control)

Roast makes workflow steps declarative and reproducible through YAML, prompt files, conventions, and saved execution state.

“Roast workflows can be version-controlled, tested, and integrated into development workflows easily.”article, stated

`durable-execution` (execution reliability)

Session replay lets workflow authors resume from intermediate steps rather than rerun expensive AI operations.

“This dramatically speeds up workflow development by eliminating the need to rerun expensive AI operations over and over again.”article, stated

`eval-driven-development` (developer productivity)

Roast workflows grade tests, generate tests based on coverage feedback, run tests, and verify changes before completing.

“* Test generation that adapts based on coverage feedback”article, stated

“It will iteratively fix type errors, run tests, and ensure everything passes—something that would be nearly impossible with pure deterministic automation.”article, stated

Tradeoffs

Blend deterministic steps and AI steps.

Chose Hybrid workflows with scripts, Ruby steps, prompts, and Claude Code over Pure agentic automation or pure deterministic automation.

Rationale: AI accelerates unclear steps, while deterministic code can replace AI once the team understands the step well enough.

“Being able to trivially combine a simple deterministic sed with a high powered "do something like this" agent, and jump back and forth between the tools to ensure each one is crafting the input to the next step to give that tool the best chance of success is absurdly powerful.”article, stated

“As I understand the problem space better, it's very easy to drop the AI step for a deterministic one that always works.”article, stated

Use convention over configuration.

Chose workflow.yml plus prompt files over Highly custom framework configuration for every workflow.

Rationale: Shopify wanted developers to get started quickly with predictable conventions and step formats.

“Roast follows Ruby on Rails' philosophy of convention over configuration. Simply create a workflow.yml file and corresponding prompt files, and you're ready to go:”article, stated

Gotchas (what bit them)

Free-roaming coding agents accumulate unreliability on large codebases.

Root cause: Nondeterministic agents lack guardrails when given broad access to millions of lines of code.

Resolution: Break work into structured workflow steps that constrain AI autonomy and interleave deterministic tools.

“AI agents need help staying on track, and work much better when you break down complicated prompts into discrete steps.”article, stated

“Non-determinism is the enemy of reliability.”article, stated

Expensive AI workflow development can be slow without replay.

Root cause: Debugging late workflow steps would otherwise require rerunning earlier expensive AI operations.

Resolution: Save workflow executions and resume from any step.

“One of Roast's killer features is session replay. Every workflow execution is automatically saved, allowing you to resume from any step:”article, stated

“This dramatically speeds up workflow development by eliminating the need to rerun expensive AI operations over and over again.”article, stated

Evaluation

Shopify reports internal use cases and qualitative workflow development benefits rather than a single benchmark suite.

Metrics: Thousands of test files analyzed. · A dozen engineers contributed features and workflows to Roast on GitHub. · Roast is distributed as a Ruby gem.

“Test quality at scale: Our engineers have analyzed thousands of test files, automatically identifying and fixing common antipatterns and significantly increasing test coverage across the board.”article, stated

“As a result, a dozen Engineers have already contributed features and workflows to Roast on GitHub.”article, stated

Outcomes

Roast was open-sourced as a workflow orchestration framework.
Shopify engineers used it for test quality, Sorbet type-safety automation, proactive SRE monitoring, competitive intelligence, and code-history research.
Roast workflows can be version-controlled, tested, replayed, and integrated into development processes.

The source reports thousands of analyzed test files, millions of lines of code context, and a dozen engineers already contributing workflows/features.

“Since deploying Roast internally, we've seen some fascinating use cases emerge:”article, stated

“Roast is more than just a tool, it's a new paradigm for working with AI in software development. We invite you to:”article, stated