Scaling Agentic Workflows with Temporal Cloud: Platform Engineering for Production LLM Systems

openai · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Durable workflow orchestration for production AI systems

OpenAI needed to move high-demand AI workflows from synchronous request-response flows into durable, observable, repeatable workflows that product teams could adopt quickly without owning Temporal infrastructure details.

Constraints:

Agentic workflows are long-running, cross ownership boundaries, and call services with different latency and failure modes.
Image generation went viral in two to three days and exceeded synchronous infrastructure capacity.
Product developers initially faced one to two weeks of setup before writing a workflow.
Workflows needed dashboards, traces, logs, launch readiness, and production health monitoring.
A small platform team had to support hundreds of namespaces and about a thousand workflow types.

Architecture

OpenAI adopted Temporal Cloud and built a platform layer around it. The platform provides managed workers, managed workflows, generated configs, SDK wrappers, self-hosted Temporal UI with SSO/RBAC and Datadog trace links, workflow/activity dashboards, launch-readiness practices, and clear ownership boundaries: product teams write workflow and activity code, while the platform team owns infrastructure, queues, workers, configs, observability, and paved-road automation.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Product workflow code"] --> B["Managed workflow platform"]
  B --> C["Temporal Cloud"]
  B --> D["Managed workers"]
  D --> A
  D --> C
  C --> E["Model and storage services"]
  C --> F["Workflow observability"]
  F --> G["Platform team"]
  G --> B
  H["Image generation workflow"] --> B

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Product workflow code`	Business-specific workflows and activities written by product teams.	`Managed workflow platform`
`Managed workflow platform`	OpenAI platform layer that generates resources, configs, workers, queues, and deployment boundaries.	`Temporal Cloud`, `Managed workers`, `Workflow observability`
`Temporal Cloud`	Durable execution substrate that stores state, history, retries, and long-running workflow progress.	`Model and storage services`, `Workflow observability`
`Managed workers`	Generated and operated worker services that run workflows and activities.	`Product workflow code`, `Temporal Cloud`
`Model and storage services`	Downstream models, storage engines, and services called by workflows with mixed latency/failure profiles.	`Temporal Cloud`
`Workflow observability`	Self-hosted UI, Datadog traces/logs, dashboards, and activity/workflow health metrics.	`Platform team`
`Platform team`	Small team that owns paved-road tooling, observability, platform controls, and Temporal adoption support.	`Managed workflow platform`
`Image generation workflow`	Viral workload moved from synchronous flow to Temporal to absorb backpressure and process image requests.	`Managed workflow platform`

Data flow

A product workflow is written as business logic with decorated workflows and activities. The managed-worker tooling asks for name, owner, tier, namespace, and retention, then generates worker service specs, SDK packages, images, build files, and workflow configs. The platform operator spins up resources, Temporal persists workflow state and retries calls into models/storage/services, and the self-hosted UI plus dashboards expose workflow history, logs, traces, activity health, and task growth.

3 supporting quotes

“the product team only thinks about the application code. They are only responsible for writing the workflows and activities and the rest of the part is with the uh platform team”transcript, stated

“when they click enter all of the generated specs like for worker service uh SDK packages golden images uh the whole build files work for configs automatically get generated for them.”transcript, stated

“from the workflow itself on that workflow page you can just click on that link which will just take directly take you to those traces”transcript, stated

Why this architecture

Temporal fit because OpenAI workflows are long-running, failure-prone, stateful, and cross service/team boundaries. But broad adoption required platform discipline: generated workers and workflows, observable UI, dashboards, and a boundary where product teams focused on application logic instead of infrastructure.

3 supporting quotes

“They need to be long running. They have uh code uh dependencies which can fail. They need state to be durable and they cross ownership boundaries.”transcript, stated

“the platform basically had to make temples safe path obvious but not just possible.”transcript, stated

“temporal makes durable execution possible but the platform path makes it repeatable.”transcript, stated

Models used: Temporal Cloud · Datadog · OpenAI model inference systems

Scale

Image generation went viral in two to three days.
“it went viral in like 2 three days.”transcript, stated
OpenAI reports creating about 1 billion images per week through Temporal.
“today uh we are creating like 1 billion images per week and everything is running through temporal.”transcript, stated
Temporal usage grew 60x in one year.
“it grew by 20x in Q2 and then it grew another 3x in H2. It's like 60x growth in uh in one year.”transcript, stated
The platform reached 700+ namespaces, 100+ workers, and about 1,000 workflow types.
“it's already 700 plus. There are about 100 plus workers running and thousand different workflows uh different types of workflows running on temporal cloud.”transcript, stated
Developer onboarding improved from one to two weeks to within a day.
“it used to take like one to two weeks to adopt like even start writing a workflow.”transcript, stated
A team of four manages the platform today.
“we started with like two engineers uh including myself and now today we only have a team of four managing everything.”transcript, stated

Alternatives considered (roads not taken)

Keep image generation synchronous

Rejected because: The viral load caused rate limits, rejected requests, and poor user experience.

“it was still a synchronous uh request response flow.”transcript, stated

“we started running into rate limits uh rejecting request but that was not a good user experience.”transcript, stated

Let product teams manage Temporal infrastructure directly

Rejected because: They had to configure Terraform, secrets, encryption, workers, scaling, monitoring, signals, queues, and workflow registration.

“Users had to like write terraform configure secrets encryption deploy workers scale them monitor their own signals.”transcript, stated

Expose raw Temporal Cloud UI directly

Rejected because: OpenAI self-hosted and patched the UI for SSO, RBAC, and trace/log links.

“We didn't let application developers access temporal cloud UI directly.”transcript, stated

Patterns

`durable-execution` (orchestration)

OpenAI uses Temporal Cloud for long-running workflows with durable state, retries, workflow history, and cross-service execution.

“They need to be long running. They have uh code uh dependencies which can fail. They need state to be durable and they cross ownership boundaries.”transcript, stated

“you need retries to make sure your agentic workflow succeeds.”transcript, stated

`batch-offline-processing` (serving)

Moving image generation into Temporal let OpenAI absorb backpressure from underlying infrastructure instead of rejecting requests.

“we were trying to move the synchronous flow into temporal so that we can like absorb uh all the back pressure happening from the underlying infra”transcript, stated

`observability-tracing` (operations)

The platform patched Temporal UI and dashboards with workflow history, Datadog links, traces, logs, and activity health metrics.

“we have those data dog links and traces links directly.”transcript, stated

“we provided dashboards to monitor health of our workflows and activities.”transcript, stated

`structured-output-contract` (platform)

Generated managed-worker and managed-workflow configs created a repeatable contract between product workflow code and platform-owned infrastructure.

“all of the generated specs like for worker service uh SDK packages golden images uh the whole build files work for configs automatically get generated for them.”transcript, stated

“the platform team uh the product team only thinks about the application code.”transcript, stated

Tradeoffs

Workflow ownership boundary

Chose product teams own workflows/activities, platform owns infrastructure over each product team owning full Temporal setup.

Rationale: This boundary reduced onboarding time and made 60x growth manageable.

“They are only responsible for writing the workflows and activities and the rest of the part is with the uh platform team here.”transcript, stated

“without all of this it was not possible to like uh get to the state like and launch uh product use cases.”transcript, stated

UI/control strategy

Chose self-hosted and patched Temporal UI over direct access to Temporal Cloud UI.

Rationale: Self-hosting enabled OpenAI-specific SSO, RBAC, trace links, log links, and workflow observability integration.

“what self-hosting the UI provided us was better SSO controls, better arbback policies”transcript, stated

“we were able to like patch the UI directly”transcript, stated

Image generation flow

Chose asynchronous durable Temporal workflows over synchronous request-response.

Rationale: Temporal absorbed backpressure when image generation demand exceeded infrastructure capacity.

“we were trying to move the synchronous flow into temporal so that we can like absorb uh all the back pressure happening from the underlying infra”transcript, stated

Gotchas (what bit them)

Durable execution adoption was too slow without a paved road.

Root cause: Developers had to manage Terraform, secrets, encryption, workers, scaling, monitoring, and signals before writing workflows.

Resolution: Build managed-worker and managed-workflow tooling that generates configs and deploys resources from minimal inputs.

“Users had to like write terraform configure secrets encryption deploy workers scale them monitor their own signals.”transcript, stated

“the tool basically made the infrastructure path repeatable for them”transcript, stated

A healthy worker does not imply a healthy workflow.

Root cause: Workflows can be stuck, tasks can grow, or activities can fail even when worker health looks fine.

Resolution: Provide workflow/activity dashboards and drilldowns by namespace and workflow type.

“it might seem like your worker is healthy but your workflow can still be stuck?”transcript, stated

“filter down on their namespace directly their workflow type and like drill down”transcript, stated

Critical workflows require stronger platform controls as they scale.

Root cause: Different teams have different Temporal expertise and expectations, but critical workloads need consistent reliability.

Resolution: Centralize platform controls, observability, launch readiness, and repeatable workflow/worker paths.

“as like the critical workflows uh as more critical the workflows become the more important those platform uh pieces are.”transcript, stated

“what you really want is like having a platform which the users can trust and like depend on”transcript, stated

Evaluation

The platform is evaluated through adoption and operating metrics: images per week, growth rate, namespaces, worker/workflow counts, onboarding time, production readiness, dashboards, and platform team size.

Metrics: 1 billion images per week through Temporal · 60x growth in one year · 700+ namespaces · 100+ workers · About 1,000 workflow types · Onboarding from one to two weeks down to within a day · Team of four managing the platform

“today uh we are creating like 1 billion images per week and everything is running through temporal.”transcript, stated

“everything is like within a day like users can like just decorate their workflows and activities as part of writing the business logic”transcript, stated

Outcomes

Image generation moved from synchronous flow to Temporal and now processes about 1 billion images per week.
Temporal adoption grew 60x in one year.
The platform reached 700+ namespaces and about 1,000 workflow types.
Workflow onboarding dropped from one to two weeks to within a day.
A four-person platform team manages the paved-road workflow platform.

OpenAI reports 1B images/week, 60x annual Temporal growth, 700+ namespaces, 100+ workers, about 1,000 workflow types, and a four-person platform team.