Building Durable and Reliable AI Agents at Scale with Dapr Workflows

humanlayer · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Durable cloud-native AI agent infrastructure

Dapr's AI-agent work addresses a production reliability gap in agent frameworks: agents can lose state, replay expensive LLM calls, duplicate workflows, fail to recover after outages, and lack secure distributed communication when deployed across pods or services. The talk positions Dapr workflows, state management, pub/sub, access policy, conversation APIs, and OpenTelemetry as distributed-systems primitives for making autonomous agent workflows reliable at enterprise scale.

Constraints:

• Agentic workflows may contain 100 or even 1,000 steps, so replaying from the beginning after a late failure is costly.
• Production failures include cloud outages, pod crashes, network blips, LLM throttling, and unexpected model responses.
• Multi-agent systems need discovery, secure calls, and at-least-once message delivery beyond a single process.
• Existing agent frameworks often save checkpoints but leave failure detection, recovery, and duplicate-execution prevention to the application team.
• Enterprise agent systems need governance, PII handling, audit trails, retry policies, and database portability.

Architecture

Dapr Agents wraps agent applications with Dapr sidecars and workflow/state/pub-sub building blocks. A durable agent declares role, instructions, LLM provider, and a Dapr state-memory store. Dapr workflows persist workflow progress, detect failures, retry according to policy, recover from the exact failed step, and enforce one active execution per workflow instance. Dapr state management lets agents store conversation state in 30-plus databases. Dapr pub/sub lets agents discover and message each other with at-least-once delivery. Conversation APIs add LLM abstraction and PII obfuscation, access policies and trust domains govern which agents and MCP servers can talk, and OpenTelemetry traces provide audit logs and debugging visibility.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  agent-application[Agent application] --> dapr-sidecar[Dapr sidecar]
  dapr-sidecar --> dapr-workflow-engine[Dapr workflow engine]
  dapr-sidecar --> state-store[State store]
  dapr-sidecar --> dapr-pub-sub[Dapr pub/sub]
  dapr-sidecar --> conversation-api[Conversation API]
  dapr-sidecar --> access-policies-and-trust-domains[Access policies and trust domains]
  dapr-sidecar --> opentelemetry-traces[OpenTelemetry traces]
  dapr-sidecar --> dapr-management-apis[Dapr management APIs]
  dapr-workflow-engine --> agent-application
  dapr-workflow-engine --> state-store
  dapr-pub-sub --> peer-agent[Peer agent]
  peer-agent --> dapr-pub-sub
  conversation-api --> llm-provider[LLM provider]
  llm-provider --> conversation-api
  access-policies-and-trust-domains --> peer-agent
  access-policies-and-trust-domains --> mcp-server[MCP server]
  opentelemetry-traces --> zipkin-or-tracing-backend[Zipkin or tracing backend]
  dapr-management-apis --> dapr-workflow-engine

Components

Data flow

An agent process connects to its local Dapr sidecar. The sidecar coordinates workflow execution, persists workflow and conversation state to a configured state store, invokes the LLM through the configured provider or Dapr conversation API, retries failed calls, and resumes from saved progress after crashes. For multi-agent work, agents use Dapr pub/sub declarations rather than broker-specific code; Dapr routes messages to peer agents, preserves isolation through sidecars, and emits OpenTelemetry traces showing state reads/writes, agent handoffs, and workflow completion.

Why this architecture

Dapr uses distributed-systems primitives because prototype-oriented agent frameworks focus on reasoning, tool calling, and LLM abstractions while leaving production reliability to the application team. A sidecar plus workflow engine can detect failures, coordinate recovery, prevent duplicate executions, and persist state independently of the agent process. Declarative YAML component configuration also separates application code from specific databases and brokers.

Models used: OpenAI · Google Gemini · Mistral · Anthropic · LangChain agents · LangGraph agents · CrewAI agents

Scale

• The opening risk example is a 100-call agent run that loses state and must repeat those LLM calls.
  “it's going to have to go through those 100 LLM calls over and over again.”transcript, stated
• Dapr has about 9,000 Discord members and almost 5,000 individual contributors.
  “We're coming up at 9,000 Discord members. We have almost 5,000 individual contributors”transcript, stated
• Dapr state management supports more than 30 databases.
  “you can actually save all of your data in more than 30 different databases.”transcript, stated
• The Nvidia RAG-style workflow example includes 1,000 documents, 1,000 agents, 1,000 workflows, and potentially hundreds of thousands or millions of workflow instances.
  “an agent that can process 1,000 documents”transcript, stated

  “1,000 agents running 1,000 different workflows and so you have hundreds of thousands or millions even of instances of those workflows running.”transcript, stated
• The talk uses failures at step 99 of 100 and step 999 of 1,000 to illustrate late-stage replay cost.
  “workflow consists of 100 steps and it fails at step 99, you really don't want to have to replay from the beginning.”transcript, stated

  “if you're at step 999 out of 1,000, and yes, that might happen”transcript, stated
• Dapr Agents was described as stable 1.0 and already used by people in production.
  “it's stable 1.0 since yesterday.”transcript, stated

  “we've got people using it in production.”transcript, stated
• The multi-agent demo claims support for hundreds of connected agents and even 50-agent routing by an initial orchestrator.
  “you can connect a hundreds of those agents together, all working collaboratively.”transcript, stated

  “even if you have like 50 agents, um the initial orchestrating agent will decide who's the best agent to talk to initially”transcript, stated

Alternatives considered (roads not taken)

Use agent-framework checkpointing alone.

Rejected because: Checkpointing stores state but does not automatically detect failures, recover workflows, or prevent duplicate executions.

Build a custom recovery workflow engine around existing frameworks.

Rejected because: The recovery system itself would need durability, failure detection, bulk recovery, and multi-instance coordination.

Put all logic into an agent monolith.

Rejected because: The speaker recommends spreading logic across agents that each do one job, using Dapr to discover and message peers.

Write broker-specific Kafka, Pulsar, or SQS code for multi-agent communication.

Rejected because: Dapr pub/sub lets the same agent code use a YAML-declared broker and switch infrastructure with configuration.

Migrate every team to Dapr Agents immediately.

Rejected because: The roadmap also brings Dapr benefits into existing frameworks so teams can keep LangChain, LangGraph, Strands, or OpenAI Agents code where needed.

Patterns

durable-execution (infrastructure)

Dapr workflows persist execution state, detect crashes, retry failed calls, resume from the failed step, and prevent duplicate workflow instances.

async-message-orchestration (multi-agent)

Dapr pub/sub provides discovery and message delivery for distributed agents without broker-specific code, allowing asynchronous collaborative workflows.

agent-memory (state)

Dapr Agents save both workflow progress and conversation memory to the selected state store so recovered agents do not recompute previous context.

privacy-preserving-processing (governance)

The Dapr conversation API can obfuscate PII both before data enters an LLM and after responses come back.

safety-policy-enforcement (governance)

Dapr access policies and trust domains restrict which agents can communicate with other agents and MCP servers at runtime.

observability-tracing (operations)

The system emits OpenTelemetry traces for full audit logs, state reads/writes, agent handoffs, and timing across multi-agent workflows.

Tradeoffs

Use Dapr workflows for agent durability.

Chose workflow-engine recovery over checkpoint-only persistence.

Rationale: Checkpoint-only systems save state but still require teams to build failure detection, recovery, and duplicate-execution prevention.

Use Dapr component YAML for infrastructure choices.

Chose declarative database and broker bindings over SDK-specific code paths for each backend.

Rationale: The same agent code can switch state stores or pub/sub brokers by changing Dapr component configuration.

Distribute multi-agent logic across agents instead of building an agent monolith.

Chose Dapr-connected agent collaboration over one agent containing all logic.

Rationale: Separate agents can own narrower responsibilities while Dapr handles discovery, messaging, and isolation.

Integrate Dapr benefits into existing frameworks as well as Dapr Agents.

Chose extensions and integration points over forcing code migration to Dapr Agents.

Rationale: Many teams already chose LangChain, LangGraph, Strands, OpenAI Agents, or custom frameworks and still need durability primitives.

Obfuscate PII inside the LLM abstraction layer.

Chose Dapr conversation API PII protection over unfiltered model input/output.

Rationale: Enterprises may need to protect sensitive values going into hosted or self-hosted models and also sanitize model outputs before returning them.

Gotchas (what bit them)

State loss can make late-stage agent failures financially expensive.

Root cause: Without durable execution, an agent that fails near the end of a long workflow may repeat all prior LLM calls.

Resolution: Use Dapr workflows so execution state persists and the agent resumes from the failed point.

Checkpointing alone is not recovery.

Root cause: Frameworks can serialize step outputs but still lack failure detection and recovery orchestration.

Resolution: Let Dapr handle failure detection and recovery instead of making every team build a workflow engine.

Recovery systems can duplicate workflow execution.

Root cause: Multiple recovery instances may resume the same workflow twice if the framework does not coordinate active executions.

Resolution: Dapr workflows enforce one executing instance per workflow.

MCP servers and autonomous agents expand the security attack surface.

Root cause: Agents and MCP servers can hold credentials to databases, CRM systems, and APIs, enabling lateral movement after a breach.

Resolution: Apply access policies and trust domains to govern agent-to-agent and agent-to-MCP communication.

Framework-native state stores can lock teams into a narrow database set.

Root cause: Some agent frameworks support only a few default databases for saved state.

Resolution: Use Dapr's state abstraction to write agent state to 30-plus supported databases and migrate stores by configuration.

Evaluation

The talk demonstrates reliability through failure-oriented demos rather than only happy-path examples. A weather agent is killed mid-run and restarted without a second request; Dapr resumes exactly where it left off. A multi-agent demo runs two isolated agents through Dapr sidecars, pub/sub, and asynchronous workflow invocation. Zipkin/OpenTelemetry traces are inspected to verify audit logs, state get/save operations, conversation-state persistence, and step timing.

Metrics: resume from exact failed step · no re-submitted user request after restart · retry behavior for LLM or network failures · exactly-once workflow execution · at-least-once pub/sub delivery · number of supported state databases · agent discovery with four lines of code · OpenTelemetry audit coverage · conversation state and workflow state persistence · ability to connect hundreds of agents

Outcomes

• Dapr Agents provides durable agents that can resume after abrupt process failure
• Dapr workflows provide failure detection, recovery, retry policy support, and exactly-once workflow execution
• Dapr state management lets agents store state in more than 30 databases
• Dapr pub/sub enables distributed multi-agent discovery and at-least-once messaging
• Dapr conversation APIs add provider abstraction and bidirectional PII obfuscation
• Dapr access policies and trust domains govern agent and MCP server communication
• OpenTelemetry traces provide audit logs and debugging visibility across agent workflows
• Dapr Agents reached stable 1.0 and had production users at the time of the talk

The talk gives concrete scale anchors: 100 LLM calls, 100-step and 1,000-step workflows, 1,000 documents, 1,000 agents and 1,000 workflows creating hundreds of thousands or millions of workflow instances, 30-plus databases, 9,000 Discord members, almost 5,000 contributors, retry-for-five policy examples, two-agent demo isolation, hundreds of connected agents, and stable 1.0 release timing.