Building Production AI Agents with Temporal-Based Workflow Orchestration

retool · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise agent orchestration on durable workflow infrastructure

Retool needed to launch production AI agents for enterprise users without rebuilding durability, retries, human approval, observability, state persistence, and scale infrastructure from scratch. Agents could run for minutes to hours, crash mid-task, call unreliable tools, require dangerous-action approval, and need auditability when they made bad decisions.

Constraints:

• Agents can run for minutes to hours and may crash after many prior tool/database/messaging steps.
• Context must be preserved without polluting parent agent state when sub-agents run independently.
• Tools, APIs, databases, and rate-limited services fail and need configurable retries.
• Dangerous actions may need to pause indefinitely while waiting for human approval.
• Enterprise customers need audit trails, security, governance, and state persistence from day one.
• Workflow execution cost and latency matter at Retool's scale of millions of workflow runs per day.

Architecture

Retool reused its Temporal-backed Workflows platform as the execution substrate for Retool Agents. Clients start agents through Retool, the backend kicks off a workflow run, Temporal queues work for agent workers, and the agent follows a loop of LLM call, optional human approval, tool execution, and continuation until completion. Temporal workflows represent deterministic state transitions and human-approval signal handling, while Temporal activities run nondeterministic LLM calls, tool calls, user code, status updates, billing, audit logs, and other side effects. Child workflows isolate sub-agent context. Temporal Signals implement human-in-the-loop approval, and event history provides a complete audit trail. Dedicated namespaces, task queues, and Datadog monitors keep agent workloads separate from standard Retool Workflows.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  retool-client["Retool client"] --> retool-backend["Retool backend"]
  retool-client --> agent-chat-interface["Agent chat interface"]
  retool-backend --> temporal-cluster["Temporal cluster"]
  retool-backend --> sandbox-code-executor["Sandbox code executor"]
  temporal-cluster --> agent-worker["Agent worker"]
  temporal-cluster --> temporal-workflow-state-machine["Temporal workflow state machine"]
  temporal-cluster --> temporal-activities["Temporal activities"]
  agent-worker --> temporal-workflow-state-machine
  agent-worker --> agent-execution-block["Agent execution block"]
  agent-execution-block --> temporal-workflow-state-machine
  agent-execution-block --> temporal-activities
  temporal-workflow-state-machine --> temporal-activities
  temporal-workflow-state-machine --> human-approval-signal-handler["Human approval signal handler"]
  temporal-workflow-state-machine --> child-workflow-sub-agents["Child workflow sub-agents"]
  temporal-activities --> tool-execution-layer["Tool execution layer"]
  temporal-activities --> sandbox-code-executor
  tool-execution-layer --> human-approval-signal-handler
  human-approval-signal-handler --> retool-client
  temporal-cluster --> temporal-event-history["Temporal event history"]
  temporal-cluster --> dedicated-agent-infrastructure["Dedicated agent infrastructure"]
  dedicated-agent-infrastructure --> agent-worker
  dedicated-agent-infrastructure --> datadog-monitors["Datadog monitors"]
  temporal-event-history --> datadog-monitors

Components

Data flow

A Retool client starts an agent. The Retool backend creates a workflow run and passes it to a Temporal queue. Agent workers pick up the task and execute a dynamic state machine: call the LLM, pause for human approval when required, execute tools as Temporal activities, then loop until done. If a tool action needs approval, the workflow edge registers a signal handler and waits on a promise race between an approval signal and a sleep timer. Tool failures use activity retries with configurable backoff. Sub-agents run as child workflows with their own event history, timeout policy, and retry policy. When the workflow completes, Retool returns a human-readable response to the agent chat UI.

Why this architecture

Retool already had Temporal-based workflow infrastructure that matched the hard parts of production agents: durable execution, retryable tools, human approval, observability, state persistence, and long-running loops. Reusing those primitives let a small team avoid rebuilding state persistence, crash recovery, retries, human-loop infrastructure, and auditability, while preserving go-to-market velocity. Static Retool Workflows can use a V3 preplanned DAG for speed/cost, but dynamic agents use the same conceptual stack with a V2-like per-iteration activity grouping because the loop can continue until the task is complete.

Models used: LLM agent model

Scale

• Retool Workflows has thousands of customers and more than 10 million workflow runs per day.
  “we have a thousands of customers and we have run over 10 million workflow runs per day.”transcript, stated
• The workflow example has over 30 blocks.
  “this is an example of a real customer workflow with over 30 blocks.”transcript, stated
• Temporal activity overhead is estimated at 50 to 100 milliseconds per activity.
  “the Temporal overhead is about 50 to 100 milliseconds per activity”transcript, stated
• A 10-block workflow can spend almost a second just on activity-transition overhead.
  “this can add up to almost a second just for 10 blocks of workflow logic.”transcript, stated
• The V3 engine example reduces 30 activities to six.
  “We go from 30 activities in V1 to six, we get five times fewer activities”transcript, stated
• The optimization is described as almost eight times faster and five times cheaper.
  “it's almost eight times faster, and in temporal prices, it's five times cheaper.”transcript, stated
• Retool estimates more than $9 million a year in savings from the workflow-engine optimization.
  “that saves us over $9 million a year”transcript, stated
• Retool estimates 735 years of saved latency across customers, while acknowledging the example is contrived.
  “it's a little bit contrived, but we have 735 years of saved latency for customers overall.”transcript, stated
• Human approval was simplified to about 50 lines using Temporal Signals.
  “we did it in one and we did it in 50 lines with Temporal Signals.”transcript, stated
• Approval timers can be set to two minutes, weeks, months, or years.
  “So this timer just gives us like gives the user 2 minutes to respond, but you can theoretically set this to 2 minutes, weeks, months, even years”transcript, stated
• Retool started agents in early 2025 and launched in mid-2025.
  “we actually started work on this like early 2025, and we were able to launch Retool Agents mid-2025”transcript, stated
• Retool advertises over 200 integrations for enterprise agents.
  “over 200 integrations to any service, enterprise ready, secure by default.”transcript, stated

Alternatives considered (roads not taken)

Build an in-house Temporal-like queue with PG Boss and AWS Lambda

Rejected because: The initial in-house approach hit scaling limits around 15-minute execution, while Temporal supported weeks-long execution and enterprise guarantees.

Execute every workflow block as its own Temporal activity

Rejected because: This V1 implementation was simple but activity overhead and per-action pricing became too costly and slow at Retool's scale.

Group workflow activities only at runtime

Rejected because: The V2 eager engine reduced activity count but blocked independent branches when human approval or wait blocks paused a branch.

Use static V3 DAG preplanning for agent execution

Rejected because: Static DAG optimization fits Retool Workflows, but agents are dynamic state machines that can run until completion and are closer to V2 activity grouping.

Build human-in-the-loop infrastructure from scratch

Rejected because: A custom implementation would require polling, webhooks, state, timeouts, crash recovery, and concurrent approvals; Temporal Signals gave Retool the same capability in about 50 lines.

Run agents on the same Temporal namespace and task queues as regular workflows

Rejected because: Retool separated agent infrastructure so agent workloads could scale independently and use different timeout/execution profiles.

Patterns

durable-execution (orchestration)

Retool uses Temporal workflows and queues so long-running agent loops survive crashes, preserve state, and resume execution after approval or failure recovery.

tool-use-orchestration (agent-loop)

The agent loop is implemented as workflow state transitions around activities for LLM calls and tool calls, looping until the task is complete.

human-in-the-loop-gate (approval)

Dangerous agent actions use Temporal Signals and workflow conditions to pause, wait for approval, and resume from any device.

observability-tracing (operations)

Temporal event history acts as an execution trace that records agent decisions, tool calls, results, and failure points for debugging and audit.

agent-memory (context)

Retool uses child workflows for sub-agents so sub-agent event history, timeout policy, and retry policy do not pollute parent agent context.

fallback-degradation (resilience)

Tool failures are handled with automatic configurable activity retries and backoff rather than failing the entire agent run immediately.

latency-optimization (workflow-engine)

Retool's V3 static DAG preplanning reduces Temporal activity count and enables independent branches to run in parallel instead of blocking on human-approval branches.

cost-aware-routing (workflow-engine)

The workflow engine groups blocks into fewer Temporal activities when retry/async/start boundaries permit, reducing per-action cost and latency.

Tradeoffs

Core orchestration substrate

Chose Temporal for durable workflow execution over In-house PG Boss plus AWS Lambda queue.

Rationale: Temporal gave Retool durability, timeouts, retries, concurrency, state persistence, lower operational burden, and faster go-to-market than building infrastructure itself.

Workflow engine activity allocation

Chose V3 static DAG preplanning for workflows over V1 one-block-one-activity and V2 runtime-only eager grouping.

Rationale: Preplanning uses the static DAG to reduce activities and preserve parallel branch execution while respecting retry/async boundaries.

Agent execution model

Chose Dynamic state machine with per-iteration activity grouping over Static DAG preplanning for each agent run.

Rationale: Agent loops can continue indefinitely and choose tools dynamically, so Retool keeps the same Temporal concepts but uses a V2-like model for agent iterations.

Dangerous action approval

Chose Temporal Signals and workflow conditions over A custom polling/state/timeout/crash-recovery stack.

Rationale: Signals let workflows pause without resources and resume when approved, removing weeks of custom approval infrastructure.

Sub-agent context management

Chose Child workflows for sub-agents over Sharing sub-agent state inside the parent event/context history.

Rationale: Child workflows give sub-agents isolated event histories, timeouts, and retries, reducing parent-context pollution.

Agent infrastructure isolation

Chose Dedicated namespaces, task queues, and monitors for agents over Sharing standard workflow infrastructure directly.

Rationale: Agents need independent scaling and longer execution/timeout profiles, so they should not compete with normal Retool Workflows.

Gotchas (what bit them)

Agent crashes can erase multi-step progress without durable state.

Root cause: An agent may be many steps into a task with database queries and Slack messages already sent when the worker crashes.

Resolution: Run the agent loop on Temporal so workflow state and event history survive failures.

Naive one-block-one-activity workflow execution becomes expensive and slow.

Root cause: Temporal adds 50-100ms overhead per activity and charges per action, so many activities multiply latency and cost.

Resolution: Group blocks into activities where retry, start, and async-completion boundaries allow it.

Runtime activity grouping can block independent branches.

Root cause: The V2 eager engine groups while iterating at runtime, so a human-approval/wait block can hold branches that do not depend on it.

Resolution: Pre-plan the static workflow DAG in V3 so independent activity nodes can execute in parallel.

Agents cannot always use the same static-DAG optimization as workflows.

Root cause: Agent execution is a dynamic state machine that can theoretically continue forever rather than a known DAG.

Resolution: Use the same Temporal primitives, but execute agents closer to the V2 model with each iteration of tool calls grouped into one activity.

Human approval is more than a button.

Root cause: Scratch implementations need polling, webhooks, state, timeouts, crash recovery, and concurrent approval handling.

Resolution: Use Temporal Signals with workflow conditions and a promise race between approval and a timer.

Agent workloads can interfere with regular workflow workloads if they share infrastructure.

Root cause: Agents have longer execution windows and timeout policies than standard workflows.

Resolution: Separate Temporal namespaces, task queues, and Datadog monitors for agent infrastructure.

One-at-a-time human supervision will not scale to future agent volumes.

Root cause: Future AI agents may outnumber humans and produce hundreds of thousands of tasks and approval decisions.

Resolution: Design for guardrails, crashes, retries, and a transition from human-in-the-loop toward humans-on/out-of-the-loop.

Evaluation

The talk evaluates the architecture through production-scale workflow metrics and operational reasoning rather than a formal model benchmark. Retool compares V1, V2, and V3 workflow engines on activity counts, parallelism, latency, and Temporal cost. It uses Temporal event history for agent execution traces, Datadog monitors for dedicated agent infrastructure, and workflow semantics to reason about crash recovery, retries, human approval, sub-agent context isolation, and status queries.

Metrics: Activity count per workflow version. · Temporal overhead per activity. · Workflow execution latency. · Temporal per-action cost. · Workflow runs per day. · Annual cost savings. · Saved latency across customers. · Human approval code size and wait duration. · Agent-specific namespace/task-queue/monitor isolation. · Event-history trace of tools called, results, and failure points.

Outcomes

• Retool reduced the contrived 30-block workflow example from 30 Temporal activities to six.
• The V3 workflow engine is described as five times fewer activities, almost eight times faster, and five times cheaper.
• At Retool scale, the workflow optimization is estimated to save over $9 million per year and 735 years of aggregate customer latency.
• Retool reused Temporal Workflows infrastructure to build Retool Agents in weeks rather than months.
• Retool started agent work in early 2025 and launched Retool Agents in mid-2025.
• The platform supports enterprise agent building with a visual builder, 200+ integrations, and secure-by-default positioning.

Retool reports thousands of customers and over 10 million workflow runs per day. The talk cites 50-100ms activity overhead, a 10-block workflow adding almost a second of overhead, a 30-activity to six-activity optimization, $9M/year estimated savings, 735 years of saved latency, human approval in 50 lines, 2-minute-to-years approval timers, and over 200 integrations.