Scaling AI Agents in Production: Building and Operating Hundreds of Autonomous Agents

datadog · Tech · 2026 · source Agentic: yes · promotional bias: med · summary-only (no fetched source) · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production observability and autonomous operations

Datadog needed to operate more than 100 production AI agents across SRE, developer, and security workflows while preparing to scale toward thousands of autonomous agents without losing observability, evaluation discipline, agent-native system access, or operational control.

Constraints:

The source transcript was unavailable, so this extraction is summary-only.
Agents operate in high-stakes SRE, code remediation, and security investigation workflows.
Background agents may run for hours and need durability, sandboxing, and event-driven triggers.
Evaluation must cover offline tests, online behavior, and living systems that adapt to drift.
Datadog wants model- and framework-agnostic infrastructure because model leadership changes quickly.

Architecture

Datadog treats agents as production services and first-class users of its platform. The architecture combines agent-accessible APIs and MCP servers, proactive event-driven background agents, durable execution with Temporal, containerized sandboxes, offline and online evaluation loops, model/framework-agnostic harnesses, and observability for agents themselves.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Operational signals"] --> B["Bits AI SRE"]
  A --> C["Bits AI Dev"]
  A --> D["Security analyst agents"]
  B --> E["Agent-native Datadog interfaces"]
  C --> E
  D --> E
  E --> F["Durable background runtime"]
  F --> G["Evaluation system"]
  F --> H["Observability system"]
  G --> H
  H --> I["Human operators"]
  I --> A

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Operational signals`	Alerts, errors, latency issues, and security findings that initiate agent work.	`Bits AI SRE`, `Bits AI Dev`, `Security analyst agents`
`Bits AI SRE`	Autonomous SRE agent that investigates alerts and performs initial incident work.	`Agent-native Datadog interfaces`, `Evaluation system`
`Bits AI Dev`	Code remediation agent that examines errors and proposes fixes.	`Agent-native Datadog interfaces`, `Evaluation system`
`Security analyst agents`	Agents that automate checklist-driven security investigations.	`Agent-native Datadog interfaces`, `Evaluation system`
`Agent-native Datadog interfaces`	MCP server, APIs, documentation formats, and agent-accessible product surfaces.	`Durable background runtime`
`Durable background runtime`	Event-driven, long-running, Temporal-backed, sandboxed execution environment for autonomous agents.	`Evaluation system`, `Observability system`
`Evaluation system`	Offline, online, and adaptive evaluation layers exposed to agents for improvement.	`Observability system`
`Observability system`	Monitoring, debugging, and operational telemetry for agent behavior.	`Human operators`
`Human operators`	Engineers and security staff who receive investigations, fixes, and escalation context.	`Operational signals`

Data flow

Alerts, errors, security signals, and platform events trigger specialized agents such as Bits AI SRE, Bits AI Dev, and security analysts. Agents use agent-friendly APIs, MCP tools, documentation, and Datadog data to investigate, propose fixes, or automate checklist-driven workflows. Execution is monitored through observability and evaluated through offline, online, and adaptive evaluation systems.

3 supporting quotes

“The company deployed multiple production agents including Bits AI SRE for autonomous alert investigation, Bits AI Dev for code generation and error fixes, and security analysts for automated security investigations.”summary, stated

“They have released an MCP server for Datadog, providing a structured entry point for agents to interact with their platform.”summary, stated

“Datadog emphasizes having three layers of evaluation: offline evaluation for testing before deployment, online evaluation for monitoring live performance, and a living, breathing evaluation system”summary, stated

Why this architecture

Datadog's core lesson is that model intelligence is no longer the main bottleneck; production success depends on operational excellence. The system therefore emphasizes agent-native access, proactive background execution, durable orchestration, sandbox isolation, observability, and evaluation rather than a single model or chat interface.

3 supporting quotes

“The case study emphasizes that intelligence is no longer the bottleneck—operational excellence and proper LLMOps practices are now the critical factors for successful agent deployment at scale.”summary, stated

“the winning agents are those that run in the background for hours, handling tasks without requiring human babysitting.”summary, stated

“The speaker argues strongly for avoiding lock-in to specific models or frameworks.”summary, stated

Models used: foundation models · long-context models · AI agents

Scale

Datadog reports operating more than 100 AI agents in production.
“operating over 100 AI agents in production while preparing infrastructure to scale to thousands more.”summary, stated
Datadog is preparing infrastructure to scale to thousands of agents.
“building over 100 AI agents in production and preparing to scale to thousands more.”summary, stated
The SRE agent targets 2 AM alert investigation before humans arrive.
“preventing engineers from being woken up at 2 AM for incidents.”summary, stated

Alternatives considered (roads not taken)

Reactive chat-first agents

Rejected because: Datadog found the strongest production value in proactive background agents that run for hours without babysitting.

“the winning agents are those that run in the background for hours, handling tasks without requiring human babysitting.”summary, stated

Human-only product interfaces

Rejected because: Datadog argues systems must be agent-friendly through APIs, MCP, documentation, and UX designed for agents as users.

“Datadog advocates for treating agents as first-class users of systems, which has significant architectural implications.”summary, stated

Lock into one model or framework

Rejected because: Model leadership changes rapidly, so Datadog keeps harnesses simple and expects rewrites as models evolve.

“Organizations should assume that whoever leads today will likely not lead tomorrow.”summary, stated

Patterns

`tool-use-orchestration` (orchestration)

Datadog exposes its platform through MCP, APIs, documentation, and agent-friendly interfaces so agents can investigate and act against operational systems.

“They have released an MCP server for Datadog, providing a structured entry point for agents to interact with their platform.”summary, stated

“The broader principle is ensuring that agents can actually consume your services. This includes basic but important steps like ensuring agents can read documentation through formats like LLM.txt and markdown support.”summary, stated

`durable-execution` (orchestration)

Datadog uses durable background execution, including Temporal, for long-running agents that survive failures and restarts.

“Datadog specifically calls out Temporal as a technology they use extensively for building durable agents. Temporal provides workflow orchestration with built-in fault tolerance, state management, and the ability to have long-running processes that survive failures and restarts.”summary, stated

“Temporal provides workflow orchestration with built-in fault tolerance, state management, and the ability to have long-running processes that survive failures and restarts.”summary, stated

`eval-driven-development` (evaluation)

Datadog requires clear evaluation before launching agents and maintains offline, online, and adaptive evaluation layers.

“The speaker delivers a strong message about evaluation: if you're launching an agent and don't know how you'll evaluate it, you shouldn't launch that agent. While experimentation is encouraged, production deployment requires clear understanding of how to observe problems and drive improvements.”summary, stated

`observability-tracing` (observability)

Datadog applies production observability to agents so autonomous behavior can be traced, monitored, debugged, and improved.

“The solution requires moving beyond thinking about agents as experimental systems to treating them as production services with full observability, monitoring, and operational rigor. This is where Datadog's core expertise in observability becomes directly relevant to their AI agent work.”summary, stated

“The speaker raises the "who watches the watchmen" problem: when you deploy an agent, you need something monitoring that agent, but then what monitors the monitoring agent?”summary, stated

Tradeoffs

Interaction model

Chose proactive background agents over chat and voice interfaces as the primary production pattern.

Rationale: Background agents can handle long-running work without human babysitting.

“the key lesson from production deployment is that the winning agents are those that run in the background for hours”summary, stated

Architecture portability

Chose model- and framework-agnostic design over deep lock-in to a current leading model or framework.

Rationale: Datadog expects model and framework leadership to keep shifting.

“avoiding lock-in to specific models or frameworks.”summary, stated

“Systems should be built with the expectation that they'll be rewritten as models evolve.”summary, stated

Gotchas (what bit them)

Working agents are not automatically autonomous production services.

Root cause: Autonomous agents need monitoring, evaluation, and operational rigor beyond a successful demo.

Resolution: Datadog treats agents as production services with observability, monitoring, and evaluation.

“building an agent that works is fundamentally different from building an agent that can operate autonomously.”summary, stated

Agent-hostile product design blocks automation.

Root cause: Many systems still require human UI interaction or lack APIs/docs that agents can consume.

Resolution: Design APIs, docs, MCP servers, and product surfaces with agents as first-class users.

“UX and design teams must explicitly consider agents as users when designing products, APIs, and applications.”summary, stated

“avoiding designs that require human interaction for tasks within UIs”summary, stated

Unsupervised background agents need durability and isolation.

Root cause: Agents can run for hours and must survive failures without executing unsafely on local machines.

Resolution: Use Temporal-style durable execution and containerized sandboxes.

“For background, long-running agents to work effectively, they typically need to be event-driven rather than constantly polling or requiring human triggers. This enables efficient resource utilization and appropriate responsiveness to relevant triggers.”summary, stated

“The speaker also emphasizes running agents in containerized sandboxes rather than on local machines. While acknowledging that developers enjoy tools like Open Claw for local experimentation, production deployment requires proper isolation and security.”summary, stated

Evaluation

Datadog evaluates agents with offline tests before deployment, online monitoring in production, adaptive living evaluation systems, and agent-accessible evaluation tooling.

Metrics: Offline evaluation before launch · Online evaluation of live performance · Adaptive evaluation for drift · Agent observability and monitoring · Operational automation coverage

“An innovative approach Datadog takes is making evaluation systems themselves accessible to agents. By exposing evaluation and improvement tools through MCP servers or other agent-accessible interfaces, you can create agents that think about and improve the evaluation loop itself.”summary, stated

Outcomes

Datadog operates over 100 production AI agents and is preparing for thousands more.
Bits AI SRE autonomously investigates alerts.
Bits AI Dev proposes code fixes for errors and latency issues.
Security analyst agents automate checklist-driven investigations.
Datadog productized some learnings into Dispatch Agents.

Summary-only evidence reports over 100 agents in production, preparation for thousands more, long-running background agents, and multiple productized agent categories across SRE, developer, and security workflows.

“Datadog has deployed several production-grade AI agents that serve real customer needs, providing concrete examples of LLMs operating autonomously in high-stakes production environments:”summary, stated

“The speaker mentions that these learnings have been encoded into a product called Dispatch Agents (dispatch agents.ai), suggesting Datadog is productizing their internal LLMOps practices and infrastructure.”summary, stated