Durable Execution

Definition

Durable execution preserves workflow progress across process failure, retries, long waits, and asynchronous events. The workflow engine records control state while nondeterministic model calls and external effects run as retryable activities. Block coordinates chat tasks with Temporal, and Instacart places LLM calls in activities while the workflow represents agent state. Block Instacart

Corpus Support

The analysis counts 93 deduplicated agentic systems (9.4%) across 8 industries. Evidence includes 49 article, 92 transcript, and 11 summary quotes; 151 are stated and 1 implied. Corpus analysis

Problem It Solves

Agent work may outlive a request, wait on people or services, and partially fail after side effects. Hex moved long-running data-agent jobs from a queue-oriented flow to Temporal, while Duolingo tracks test generation and pull-request lifecycle as a recoverable workflow. Hex Duolingo

Applicability Signals

• A run lasts longer than a normal request or worker lifetime. Hex
• Steps call flaky external services and need bounded retries. Block
• Work pauses for approval, polling, or later events. Duolingo
• Partial completion must resume without repeating every side effect. Instacart

Architecture and Variants

• Workflow/activity split: deterministic control flow owns state; model and external calls run as activities. Instacart
• Durable chat task: each message and its tool calls execute inside a persisted task lifecycle. Block
• Lifecycle workflow: generation, validation, pull request, and review continue over a long-running process. Duolingo
• Managed state machine: multi-stage generation runs through Step Functions. Guidesly

Why Teams Choose It

The pattern makes failure recovery, observability, and human pauses explicit instead of rebuilding them around queues and database flags. It also provides a stable control plane around nondeterministic inference. Hex Block

Tradeoffs

• Durable engines add operational concepts and determinism constraints beyond a simple worker queue. Instacart
• Persisting rich agent state improves recovery but can bloat workflow histories. Block
• Fine-grained activities improve retry isolation but increase orchestration overhead. Duolingo

Gotchas and Resolutions

• Carrying large analysis context in workflow history causes history growth; store bulky artifacts externally and persist references. Block
• Nondeterministic model calls cannot safely live in deterministic workflow code; isolate them as activities. Instacart
• Long histories require lifecycle controls such as continue-as-new. Block

When Not to Use It

Do not introduce a durable workflow engine for a short, idempotent request that can simply retry end to end. Hex adopted Temporal after long-running iterative work outgrew its prior queue flow; that migration cost is justified by lifecycle complexity, not by the presence of an LLM alone. Hex

Boundaries and Related Patterns

async-message-orchestration coordinates producers and consumers through messages; durable execution persists the state and progress of a workflow across those waits. They can coexist. workflow-decomposition defines bounded workflow slices, while durable execution governs their lifecycle. Tool orchestration chooses capabilities; it does not by itself guarantee recovery. Block

Operational Checklist

• Put model calls and external effects in retryable activities. Instacart
• Define idempotency, timeout, retry, cancellation, and compensation per activity. Block
• Keep large artifacts outside workflow history. Block
• Test worker loss, duplicate delivery, delayed approval, and resume behavior. Duolingo

Representative Implementations

Methodology and Limitations

This page samples 93 deduplicated systems. The corpus combines workflow engines, managed state machines, and checkpointing; their shared durable boundary is supported, but operational comparisons across engines are not. Corpus methodology