Building Reliable Production AI Agents with Durable Execution Infrastructure

temporal · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Durable execution for production AI agents

Temporal addresses the reliability problem of long-running AI agents that call LLMs, tools, APIs, and humans while needing to survive crashes, retries, outages, sleeps, and state recovery without making every application team hand-build distributed-systems machinery.

Constraints:

Architecture

Temporal separates reliability concerns from business logic through durable execution. Developers write workflows for deterministic control flow and activities for nondeterministic I/O such as LLM calls, API calls, and tools. The Temporal server stores durable workflow state backed by a database or Temporal Cloud, workers execute code in the user's environment, and clients start or interact with workflows. Activity results are captured durably, so a workflow can replay after failure and continue from the same logical point. For agents, the LLM-driven loop remains workflow code, while LLM and tool outputs are activity results. Signals, updates, queries, timers, workflow pause, continue-as-new, UI inspection, framework integrations, encryption, worker autoscaling, large-payload storage, and streaming support extend the model for production AI systems.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  temporal-client[Temporal client] --> temporal-server[Temporal server]
  temporal-client --> workflow-code[Workflow code]
  temporal-server --> worker-process[Worker process]
  worker-process --> workflow-code
  workflow-code --> activity-code[Activity code]
  activity-code --> llm-and-tool-services[LLM and tool services]
  activity-code --> temporal-server
  workflow-code --> signals-updates-and-queries[Signals updates and queries]
  signals-updates-and-queries --> human-approval-flow[Human approval flow]
  signals-updates-and-queries --> agent-framework-integrations[Agent framework integrations]
  agent-framework-integrations --> llm-and-tool-services
  temporal-server --> temporal-ui[Temporal UI]
  temporal-server --> blob-storage-pointer[Blob storage pointer]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Temporal client Starts workflows and can configure encryption before state is sent to the Temporal server. Temporal server, Workflow code
Temporal server Durably tracks workflow state, histories, timers, activity results, and orchestration metadata. Temporal client, Worker process, Temporal UI, Blob storage pointer
Worker process Runs workflow and activity code in the customer's trusted environment such as a VPC, container, Kubernetes, or laptop. Temporal server, Workflow code, Activity code
Workflow code Deterministic control-flow code that models the agentic loop, timers, handlers, and program state. Temporal client, Worker process, Activity code, Signals updates and queries
Activity code Arbitrary I/O code for LLM calls, tool calls, APIs, databases, and other nondeterministic operations. Workflow code, LLM and tool services, Temporal server
LLM and tool services External model and tool calls whose results are saved as activity results. Activity code, Agent framework integrations
Signals updates and queries Handlers that let external systems inspect, modify, or interact with running workflow state. Workflow code, Human approval flow, Agent framework integrations
Human approval flow External input pattern where workflows pause for signatures, approvals, reminders, expirations, or retractions. Signals updates and queries, Workflow code
Agent framework integrations OpenAI Agents SDK, Pydantic AI, Vercel AI SDK, BrainTrust, LangFuse, and similar integrations that keep the AI stack flexible while Temporal handles durability. LLM and tool services, Signals updates and queries, Temporal server
Temporal UI Observability surface for workflows, activities, arguments, return values, retries, and failures. Temporal server
Blob storage pointer Pass-by-reference storage path for large payloads kept in customer blob storage while Temporal tracks metadata. Temporal server

Data flow

A client starts a workflow with arguments. The Temporal server records workflow state while workers execute code in the customer environment. Deterministic workflow code controls the agentic loop and schedules activities. Activities perform nondeterministic work such as LLM calls, tool calls, or API calls; when results return, the SDK saves them to the Temporal server. If a worker crashes, another worker can replay the workflow from durable history, reuse captured activity results, and continue. External systems interact with running workflows through signals, updates, and queries. For sensitive data, the client encrypts payloads before they reach Temporal Cloud; workers decrypt and compute inside the trusted environment. Large objects can be stored in blob storage with encrypted pointers tracked by Temporal.

5 supporting quotes
“you'll have a client which is going to be sort of the declaration of go run this workflow. These are the arguments.”transcript, stated
“>> Yeah. >> Can I even use durable execution with LLMs? I think as we talked about before, that agentic loop actually is deterministic code. >> It's not the output. >> The The output of the LLMs, even if it's controlling. So so the difference with an LLM and with”transcript, stated
“whenever that result comes back what happens in the Temporal SDK is that it takes that result and it saves it out to the Temporal server.”transcript, stated
“compute on that data either. >> [gasps] >> And so that's where you bring up the worker, right? And the worker is running in your VPC, your trusted environment. And um and and that's actually where you're going to decrypt the data and then it's going to run, you know, just like it”transcript, stated
“what's going to happen is is that say that output that got generated, uh it's going to be the Temporal client is going to be configured”transcript, stated

Why this architecture

Temporal's architecture exists to make reliability a platform concern instead of application boilerplate. Deterministic workflow replay gives checkpoint-like recovery without requiring developers to write coarse checkpoint save/restore code. Activities isolate nondeterministic LLMs and tool calls so agentic systems still fit the durable model. Server/worker separation protects customer data and lets computation run in trusted environments. Integrations let teams keep their agent frameworks while using Temporal for long-running, async, crash-proof execution.

5 supporting quotes
“we have reliability over here, we have business logic over here, and those can be treated as separate concerns.”transcript, stated
“mechanism within Temporal for recovering from that, and what's important about it is that the programmer doesn't need to do extra work to do that. Of course, you can write retry logic. Of course, you can think about distributed systems. Like, you can do it. People have”transcript, stated
“>> Absolutely, you don't need to know about it and you actually also don't need to think about writing that save code in most situations. I'll say there is this like I said, you can do the continuous new, you can do checkpoints for things that are going to run for a long time and generate state for”transcript, stated
“the AI ecosystem do you want to bring in and use to build your agentic system, right? And let us handle the durability. Let us make it reliable. Let us give you the ability to use these sort of primitives to um, to send messages and queries and kind of ask the agent,”transcript, stated
“should you do next?" >> Yeah, that it's it's really just separating different concerns. So, the Temporal state is your program state. I mean, another way to think about it is, you know, whatever your stack and and memory state is is effectively protected”transcript, stated

Models used: LLMs invoked as activities · Agent frameworks such as OpenAI Agents SDK and Pydantic AI · Coding agents guided by a Temporal skill

Scale

Alternatives considered (roads not taken)

Hand-written retry and recovery scripts

Rejected because: Manual recovery scripts become messy and do not scale well to teams or complex business logic.

“Can I write a job that goes through at the end of the day and figures out where things broke and then writes a script to to fix all those?”transcript, stated
“right? It's like it's kind of doable. Do I want to do that? Is that robust? Is that reliable? Does that scale to a team and complex functionality? What if there's like more business logic? What if it's not just debit credit? What if there's, you know, some uh checks”transcript, stated

Event-driven architecture with queues and compensation logic

Rejected because: Teams can cover edge cases with queues, dead letters, and compensation, but durable execution simplifies the resulting code.

“You can cover all the edge cases. You can have the dead letter queues and you can have the compensation logic and you can do all of that, right? Like, you can do it. With a durable execution abstraction, that code just becomes really simple. It gets cleaned”transcript, stated
“you can do all of that, right? Like, you can do it. With a durable execution abstraction, that code just becomes really simple. It gets cleaned up. You make sure that that activity code on a piece-by-piece basis is item potent in terms of the operations on the database. That basically means”transcript, stated

Coarse checkpoint-based recovery

Rejected because: Checkpoints require explicit save/restore logic and can be poorly placed for tool calls, human interactions, or concurrent programs.

“machine learning training, for example. And, you know, checkpoints are okay, right? But they are um coarse-grained, generally speaking. So, if you are in the middle of say, um multiple tool calls or maybe you have a human in the loop interaction, it's not clear that that point in the code is going”transcript, stated
“you're still writing a function to take all my state, save it out, and bring it back. And then you're getting more coarse-grained uh recovery. >> So, I thought this was a form of checkpointing. How is it different? >> Yeah, so I think you can define the”transcript, stated

Store large AI payloads directly in Temporal server

Rejected because: Media files or large outputs can stay in customer blob storage while Temporal tracks encrypted metadata and pointers.

“you may not want to send them through the Temporal server and store them there, uh even if they're encrypted. You you say, "You know what? That thing lives in S3, right?" Or or something like that. But, I want Temporal to understand it. I want Temporal to”transcript, stated
“Exactly. Exactly. It's it's Yes, at the end of the day, we know we're talking about pass by reference rather than pass-by-value. >> Yeah. >> So, we're doing that. We're doing streaming, which is, you know, another thing that's really important in agentic use case. You actually have pretty”transcript, stated

Patterns

durable-execution (reliability)

Temporal captures workflow history and activity results so long-running agents can recover from crashes, sleeps, and failures without developers writing explicit checkpoint logic.

“durable execution basically just means that your software does what it's supposed to do.”transcript, stated
“it saves it out to the Temporal server. So, that's where it's captured durably”transcript, stated

tool-use-orchestration (architecture)

The agentic loop calls LLMs and tools through activities while workflow code remains deterministic.

“it's iterating over calling the LLM, calling tools, calling the LLM, calling tools until it's done.”transcript, stated
“those activities, which are say tool calls, maybe calling an API or calling an LLM”transcript, stated

human-in-the-loop-gate (control)

Workflows can pause for signatures, approvals, reminders, expiration, or retraction while compute resources are reclaimed.

“The other case that's pretty valuable is for human in the loop.”transcript, stated
“we want to be able to reclaim the resources, which are basically in the temporal, those are going to be cached compute resources”transcript, stated

observability-tracing (monitoring)

Temporal UI exposes workflows, activities, arguments, return values, retries, and failures during development and operations.

“And then the other thing that I encourage folks to do is to use the Temporal UI. So that when the if you start up the CLI, if you go to the cloud, it'll just be there.”transcript, stated
“you can see the activities, you can dive in, you can see the arguments, the return values, uh you can see any sort of retries, you can see failures.”transcript, stated

privacy-preserving-processing (security)

Temporal Cloud does not see application data because clients encrypt before sending and workers decrypt inside trusted environments.

“we actually don't see any of your data. Okay? Any of your application data.”transcript, stated
“the Temporal client, to encrypt everything before it gets sent to the temporal server”transcript, stated

canary-rollout (deployment)

The summary describes worker versioning and blue-green style deployment for rolling out new agent behavior without disrupting running workflows.

“Worker versioning enables sophisticated deployment strategies like blue-green deployments”summary, stated
“existing long-running workflows can continue on the old version while new workflows start on the new version.”summary, stated

Tradeoffs

Separate workflow and activity code

Chose Deterministic workflows plus arbitrary I/O activities over Putting all agent logic and I/O in one nondeterministic function.

Rationale: Workflow replay stays deterministic while LLM and tool outputs are captured as durable activity results.

“So, activities are um doing IO and basically they're arbitrary code. What makes durable execution work is that the workflow code is uh a um restricted program model.”transcript, stated
“I think as we talked about before, that agentic loop actually is deterministic code. >> It's not the output. >> The The output of the LLMs, even if it's controlling.”transcript, stated

Use durable execution instead of event-driven plumbing

Chose Temporal's durable abstraction over Queues, dead-letter queues, compensation logic, and manual retries everywhere.

Rationale: The same reliability can be built by hand, but the durable abstraction keeps application code simpler.

“You can have the dead letter queues and you can have the compensation logic and you can do all of that”transcript, stated
“With a durable execution abstraction, that code just becomes really simple.”transcript, stated

Keep Temporal state separate from agent memory

Chose Program state protected by Temporal, agent framework state handled by the AI stack over Treating Temporal as the entire memory layer for personalization and multi-agent behavior.

Rationale: Temporal tracks execution path and durable program state; agent frameworks still own memory, guardrails, and multi-agent composition.

“separating different concerns. So, the Temporal state is your program state.”transcript, stated
“the agent framework is about making it easy to put together, you know, multiple agents and guardrails”transcript, stated

Encrypt application data before it reaches Temporal server

Chose Server orchestration over encrypted state plus customer-run workers over Letting the Temporal server see and compute on customer data.

Rationale: This supports enterprise privacy but means compute must happen in trusted worker environments.

“The Temporal server, and certainly like in in Temporal Cloud, we actually don't see any of your data.”transcript, stated
“And the worker is running in your VPC, your trusted environment. And um and and that's actually where you're going to decrypt the data and then it's going to run, you know, just like it would run ordinarily >> Mhm.”transcript, stated

Gotchas (what bit them)

LLM nondeterminism must not be placed inside workflow code.

Root cause: Temporal replay depends on workflow determinism, while LLM outputs and API calls are inherently nondeterministic.

Resolution: Put LLM calls, API calls, and tool calls in activities and save their results durably.

“activities are um doing IO and basically they're arbitrary code. What makes durable execution work is that the workflow code is uh a um restricted program model. And in particular, the restriction that we have on that is to say if something goes wrong and we rerun that code provided we give it the”transcript, stated
“Or or or or [snorts] or or an LLM or an LLM, um which is sort of the obvious case for for an activity”transcript, stated

Checkpointing is too coarse for branching and concurrent agent programs.

Root cause: Tool calls, human interactions, and parallel branches make it hard to choose clean checkpoint points and write recovery code.

Resolution: Use durable history and deterministic replay, with continue-as-new only at deliberate history compaction points.

“if you are in the middle of say, um multiple tool calls or maybe you have a human in the loop interaction”transcript, stated
“getting clean checkpoints across a uh sort of concurrent program like that basically doesn't really work.”transcript, stated

Workflow histories can grow too large for indefinitely running agents.

Root cause: Long-lived agents accumulate incremental state deltas and history over time.

Resolution: Use continue-as-new at suitable loop boundaries to create a full checkpoint and compact history.

“It's just gotten too big of a history." And oh, and and it's a good point for me to do it, right? And in some cases like that, top of that uh say say it's a chat, you know, the top of that loop where you say, "Okay, now we're pausing the agent.”transcript, stated
“in Temporal language it's called continue as new, and now we basically do a a you know, a full checkpoint.”transcript, stated

Temporal Cloud cannot compute on encrypted customer data.

Root cause: The server can orchestrate encrypted state but lacks the customer keys and therefore cannot run business logic over decrypted payloads.

Resolution: Run workers in the customer's VPC or trusted environment where data is decrypted and code executes.

“And the reason for that is because you'll end up configuring in in production, you'll configure um the client, the Temporal client, to encrypt everything before it gets sent to the temporal server and then you have the keys, we don't have the keys. >> Yeah.”transcript, stated
“if you can't see the data? Well, actually you can't compute on that data either.”transcript, stated

Large media or document payloads should not always be passed by value through Temporal.

Root cause: Large AI payloads can be inefficient or inappropriate to store in the Temporal server even when encrypted.

Resolution: Store objects in customer blob storage and pass encrypted pointers or metadata through Temporal.

“We are working on some um support for larger data sizes, large payload storage, right? So, if you have something that is maybe again like some of these these media files, you may not want to send them through the Temporal server and store them there, uh even if they're encrypted.”transcript, stated
“put that in your blob storage, remember the path to it, and then encrypt that, put that in Temporal server”transcript, stated

Evaluation

Temporal's evaluation and observability story centers on deterministic replay and operational inspection: workflow state is replayable, activity results are durable, the UI shows workflow execution, arguments, return values, retries, and failures, developers can fix code after a crash and restart workers, and integrations with BrainTrust and LangFuse connect durable execution to AI observability tools.

Metrics: Workflow completion despite crashes or failures · Activity retries and failures · Replay determinism · Worker crash recovery from the failed point · Workflow, activity, argument, and return-value inspection in the Temporal UI · Region or worker recovery behavior · Framework observability through BrainTrust and LangFuse integrations

4 supporting quotes
“your local environment, you'll get a URL, you click that, you open it up, and basically what you can then see is you can see the program executing. So you can see um you know, the the the the workflow, you can see the activities, you can dive in, you can see the arguments, the return values, uh”transcript, stated
“the the workflow, you can see the activities, you can dive in, you can see the arguments, the return values, uh you can see any sort of retries, you can see failures. And actually one of the cool things that I encourage folks to do this this kind of blew my mind when I when I first um saw it is”transcript, stated
“So what you can do now is you can just go, you can fix that error, >> Oh. >> save the file, >> where you're going.”transcript, stated
“We have integrated with um, BrainTrust on the observability side. We've integrated with LangFuse.”transcript, stated

Outcomes

Concrete scale includes multi-region cloud replication, workflows that can run forever, timers such as every half hour or daily at 3:00 p.m., human pauses of 6 or 24 hours, and named production users including OpenAI Codex on the web, Replit, and Lovable. Quantitative success metrics are not reported.

4 supporting quotes
“At Temporal, we talk about making software crash-proof. And that means that whatever it is, program starts at the beginning, and it's going to run to the end regardless of what happens with failures, which are particularly common in the cloud, right?”transcript, stated
“in addition to OpenAI's Codex on the web, we have um Replit and and lovable”transcript, stated
“OpenAI agents SDK, um Pydantic AI, um um uh AI SDK by Vercel, those are ones that have been released.”transcript, stated
“And let us handle the durability. Let us make it reliable. Let us give you the ability to use these sort of primitives to um, to send messages and queries and kind of ask the agent, "What are you doing right now, right?”transcript, stated