Building and Scaling AI Agents in Production for DevSecOps Automation

datadog · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: DevSecOps automation

Datadog needed production agents that could autonomously investigate operational, development, and security work inside its observability platform, while preparing the company to scale from the first hundred agents toward thousands more.

Constraints:

agents must operate on real DevSecOps systems rather than only chat with humans
background agents need durable long-running execution, state, storage, and safe isolation
agent behavior must be observable and evaluated before and after launch
interfaces, documentation, APIs, and UX need to become usable by agents as first-class customers
model and framework leadership changes quickly, so the harness should remain rewritable and model-agnostic

Architecture

Datadog's agent program combines production agents for SRE, development, and security work with an agent-native platform posture. Bits AI SRE investigates alerts, Bits AI Dev proposes operational code fixes from errors and latency issues, and Security Analyst automates checklist-style security investigations. The supporting architecture emphasizes agent-friendly APIs and docs, an MCP server entry point, event-driven background workflows, Temporal-backed durable execution, containers/sandboxes with storage and filesystems, evaluation and monitoring loops, simple rewritable harnesses, memory systems that survive model changes, and communication patterns for human-agent and agent-agent collaboration.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  datadog-platform[Datadog platform] --> bits-ai-sre[Bits AI SRE]
  datadog-platform --> bits-ai-dev[Bits AI Dev]
  datadog-platform --> security-analyst-agent[Security Analyst agent]
  datadog-platform --> agent-friendly-interfaces[Agent-friendly interfaces]
  agent-friendly-interfaces --> mcp-server[MCP server]
  mcp-server --> agent-harness[Agent harness]
  agent-harness --> durable-workflow-runtime[Durable workflow runtime]
  durable-workflow-runtime --> bits-ai-sre
  durable-workflow-runtime --> bits-ai-dev
  durable-workflow-runtime --> security-analyst-agent
  durable-workflow-runtime --> container-sandbox[Container sandbox]
  agent-harness --> evaluation-and-monitoring[Evaluation and monitoring]
  agent-harness --> memory-systems[Memory systems]
  memory-systems --> model-and-framework-layer[Model and framework layer]
  agent-harness --> model-and-framework-layer

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Datadog platform`	Observability and DevSecOps platform where alerts, latency issues, errors, and security signals originate.	`Bits AI SRE`, `Bits AI Dev`, `Security Analyst agent`, `Agent-friendly interfaces`
`Bits AI SRE`	Production SRE agent that autonomously investigates alerts to reduce wake-ups and incident investigation toil.	`Datadog platform`, `Durable workflow runtime`, `Evaluation and monitoring`
`Bits AI Dev`	Production development agent that inspects service errors and latency issues and proposes code fixes inside Datadog.	`Datadog platform`, `Durable workflow runtime`, `Container sandbox`
`Security Analyst agent`	Production security agent that automates checklist-driven investigations for concerning security alerts.	`Datadog platform`, `Durable workflow runtime`, `Evaluation and monitoring`
`Agent-friendly interfaces`	APIs, MCP server, documentation, markdown surfaces, and UX decisions designed so agents can use Datadog as customers.	`Datadog platform`, `MCP server`, `Agent harness`
`MCP server`	Datadog entry point that lets external or internal agents interact directly with the platform.	`Agent-friendly interfaces`, `Agent harness`
`Agent harness`	Simple, rewritable orchestration layer around model calls, tools, memory, and workflow execution.	`MCP server`, `Memory systems`, `Durable workflow runtime`, `Model and framework layer`
`Durable workflow runtime`	Temporal-style orchestration for long-running, event-driven background agents that survive interruptions.	`Agent harness`, `Bits AI SRE`, `Bits AI Dev`, `Security Analyst agent`, `Container sandbox`
`Container sandbox`	Contained execution environment with storage and filesystems so agents do not run directly on local machines.	`Durable workflow runtime`, `Bits AI Dev`
`Evaluation and monitoring`	Offline evals, online evals, production monitoring, and improvement tools that observe agents and drive iteration.	`Bits AI SRE`, `Security Analyst agent`, `Agent harness`, `MCP server`
`Memory systems`	Stores organizational and task learnings so they can move across future models and frameworks.	`Agent harness`, `Model and framework layer`
`Model and framework layer`	Swappable model and framework choices kept decoupled from the agent harness.	`Agent harness`, `Memory systems`

Data flow

A Datadog alert, error, latency issue, or security signal enters the platform and is routed to the relevant agent, such as Bits AI SRE, Bits AI Dev, or Security Analyst. The agent interacts with Datadog through agent-friendly APIs, documentation, and MCP surfaces, runs as a background event-driven workflow rather than a babysat chat session, and executes inside durable, contained infrastructure such as Temporal-backed workflows and containers. Evaluation and monitoring observe failures and drift, while memory and simple harness layers keep the learned workflow portable across future model and framework changes.

3 supporting quotes

“First is the Bits AI SRE. This is one of the first SRE automated SREs that was on the market. Um, this one really thinks a lot about what does it mean to autonomous- autonomously investigate alerts.”transcript, stated

“the agents that are winning today are going to be the agents that can run in the background. Run for hours, you know. Go and handle the tasks that you don't want to have to sit there and baby sit.”transcript, stated

“If you're launching an agent and you have don't know how you're doing eval, please don't launch that agent. Um go and try with try it out, but make sure you understand how you're going to observe if things go wrong. Make sure you understand how you're going”transcript, stated

Why this architecture

Datadog's architecture starts from the premise that useful agents are autonomous operators, not chat widgets. Agent-first APIs and docs let agents become customers of the product; event-driven durable workflows let them run for hours without babysitting; containers and sandboxes create operational boundaries; eval and monitoring are launch prerequisites; and simple, model-agnostic harnesses protect the organization from rapidly shifting model and framework winners. Memory systems carry learned operating context forward even when the model changes.

4 supporting quotes

“make sure all your interfaces are agent friendly. Uh do not go and make make it so that everything that has to be done inside your system, inside your UI, has to be done by a human.”transcript, stated

“if they're in the background running without baby sitting, you really need durable agents. Temporal is great for this. Big shout out to them. Uh we use them uh quite a lot.”transcript, stated

“today will probably not be the best one tomorrow. Don't build for one particular framework or build for one particular model. You know, we have this kind of like it's so over, we're so back situation. Um I think a few months ago people would have or just before December people would have”transcript, stated

“one of the tricks to doing this is make sure you have good memory systems so you can take the learnings uh that your agents have have kind of gathered over time in your company and move them regardless of what the model is.”transcript, stated

Models used: AI agents · Claude · Codex · multimodal models

Scale

Datadog framed the work as scaling from the first hundred or so agents toward the next thousand.
“what it really took for us at DataDog to build out the first hundred or so agents that we use and how to prepare for the next thousand.”transcript, stated
The speaker had been building AI agents and assistants for about 15 years.
“I've been building AI friends and co-workers for quite a while now, 15-ish years.”transcript, stated
Bits AI SRE targets 2:00 AM incident wake-ups by investigating alerts before engineers arrive.
“Our goal there is to make sure you're not waking up at 2:00 in the morning uh, dealing with issues.”transcript, stated
Background agents are expected to run for hours.
“the agents that are winning today are going to be the agents that can run in the background. Run for hours, you know.”transcript, stated

Alternatives considered (roads not taken)

Human-only interfaces and UIs

Rejected because: Datadog argues companies should make interfaces agent-friendly and automate tasks themselves rather than requiring humans for every system action.

“do not go and make make it so that everything that has to be done inside your system, inside your UI, has to be done by a human.”transcript, stated

Reactive chat as the primary agent modality

Rejected because: Datadog expects successful agents to run proactively in the background for long tasks rather than requiring user babysitting.

“you've heard some speakers talk about agents really started with this kind of chat interaction. That's one modality, but we're doing a lot more today.”transcript, stated

“today are going to be the agents that can run in the background. Run for hours, you know. Go and handle the tasks that you don't want to have to sit there and baby sit. So make sure you have background long-running agents. Uh and usually that means they're event-driven. And if they're in”transcript, stated

Running autonomous agents on local machines

Rejected because: The speaker recommends containers, sandboxes, storage, and filesystems for safer production execution and agent replacement.

“They shouldn't be running on your local machines. I know people love open claw and all that fun stuff. Um but if you want to do this safely, uh go and run them in containers, sandboxes”transcript, stated

Launching agents before defining evaluation

Rejected because: Datadog treats offline eval, online eval, and living evaluation systems as prerequisites for production agents.

Coupling the system to a single model or framework

Rejected because: Model leadership changes rapidly, so harnesses should be simple, rewritable, and backed by portable memory.

“whoever is going to be whoever's the best one today will probably not be the best one tomorrow. Don't build for one particular framework or build for one particular model. You know, we have this kind of like it's so over, we're so back situation. Um I think”transcript, stated

Patterns

`tool-use-orchestration` (agent-platform)

Datadog makes its APIs, docs, MCP server, and UX surfaces agent-friendly so agents can operate the platform directly.

“we have an MCP server for DataDog today. That is one of those entry ways for how a agent can be a customer of DataDog directly.”transcript, stated

“Um but really the big thing is make sure all your interfaces are agent friendly. Uh do not go and make make it so that everything that has to be done inside your system, inside your UI, has to be done by a human.”transcript, stated

`durable-execution` (runtime)

Background agents are expected to run for hours and use durable workflow infrastructure such as Temporal to survive long-running execution.

“if they're in the background running without baby sitting, you really need durable agents. Temporal is great for this. Big shout out to them. Uh we use them uh quite a lot.”transcript, stated

`eval-driven-development` (quality)

Datadog requires offline evaluation, online evaluation, and living eval systems before launching agents, and recommends making improvement tools agent-accessible.

“Um you should have offline eval, online eval, and a living breathing eval system. Things will change fast.”transcript, stated

`observability-tracing` (operations)

Agent deployment is paired with monitoring and the explicit question of how to observe agents that observe other systems.

“it's not just about building an agent that works, it's about building an agent that can operate autonomously. You know, there's a little bit of this like who watches the watchmen.”transcript, stated

`agent-memory` (learning)

Memory systems preserve organizational agent learnings so the company can switch models and frameworks without losing accumulated context.

“make sure you have good memory systems so you can take the learnings uh that your agents have have kind of gathered over time in your company and move them regardless of what the model is.”transcript, stated

`privacy-preserving-processing` (security)

Production agents are isolated in containers and sandboxes with appropriate storage and filesystems instead of running directly on local machines.

“if you want to do this safely, uh go and run them in containers, sandboxes, have appropriate storage and file systems so you can trade out agents.”transcript, stated

Tradeoffs

Agent interface posture

Chose agent-first APIs, docs, MCP, and UX over human-only UI workflows.

Rationale: Agent-native interfaces let Datadog automate internal and customer tasks directly instead of waiting for external automation to wrap human surfaces.

“You shouldn't just be designing for humans anymore. So, code an agent first. You know, Steve Yegge is famous now for his Gastown craziness.”transcript, stated

“Go and try to automate them with agents yourself. Don't wait for someone to come do it.”transcript, stated

Agent interaction model

Chose proactive background agents over reactive chat-only interactions.

Rationale: Long-running background agents can handle operational tasks without users babysitting a chat session.

“agents really started with this kind of chat interaction. That's one modality, but we're doing a lot more today.”transcript, stated

“Run for hours, you know. Go and handle the tasks that you don't want to have to sit there and baby sit.”transcript, stated

Execution safety

Chose containers and sandboxes with storage/filesystems over running agents on local machines.

Rationale: Contained environments create safer operational boundaries and allow agents to be swapped or upgraded.

“They shouldn't be running on your local machines. I know people love open claw and all that fun stuff.”transcript, stated

“go and run them in containers, sandboxes, have appropriate storage and file systems so you can trade out agents.”transcript, stated

Model strategy

Chose model- and framework-agnostic harnesses with portable memory over committing deeply to one provider or framework.

Rationale: The leading model changes quickly, so Datadog wants the harness and accumulated learnings to survive model churn.

“Don't build for one particular framework or build for one particular model.”transcript, stated

Gotchas (what bit them)

An agent can work in a demo but fail as an autonomous operator.

Root cause: Production tasks require long-running execution, recovery, monitoring, and operation without constant babysitting.

Resolution: Use background event-driven agents with durable workflow orchestration and production monitoring.

“it's not just about building an agent that works, it's about building an agent that can operate autonomously.”transcript, stated

Human-only product design blocks agent-native automation.

Root cause: Design teams may not think about agents using sites, applications, products, APIs, or docs even while developers try to automate them.

Resolution: Push UX and design teams to make interfaces, docs, APIs, and markdown surfaces usable by agents.

“The number of design teams that aren't thinking about that while the devs are just sitting there trying to make that work is quite high.”transcript, stated

Agents without evaluation lack a safe improvement loop.

Root cause: The first implementation will not work for everything, and production data drifts.

Resolution: Define offline, online, and living evaluation systems before launch and make improvement tools automatable.

“The first time you build an agent, it's not going to work for everything. Um you should have offline eval, online eval, and a living breathing eval system.”transcript, stated

Overbuilt harnesses age badly in a fast-moving model ecosystem.

Root cause: The best model and framework can change quickly, making tight coupling fragile.

Resolution: Keep harnesses simple, build them to rewrite, and store reusable learnings in memory systems.

“your agent harnesses are really the kind of important thing that you should keep very, very simple. Don't overcomplicate them. Build them to rewrite them”transcript, stated

Evaluation

Datadog advocates a three-layer evaluation loop for production agents: offline eval before release, online eval in production, and a living evaluation system that changes as data drifts. The same evaluation and improvement tools should be exposed to agents so agents can participate in their own improvement loop.

Metrics: offline evaluation · online evaluation · living breathing eval system · production monitoring · data drift

“Um you should have offline eval, online eval, and a living breathing eval system. Things will change fast. We all know the concept of data drift and all that fun stuff.”transcript, stated

“make your eval system and your improvement tools available through an MCP server or available to agents.”transcript, stated

Outcomes

Datadog had built and deployed roughly the first hundred agents and was preparing for the next thousand
Bits AI SRE, Bits AI Dev, and Security Analyst were described as production or generally available agents
Datadog established a set of operating principles around agent-first interfaces, durable execution, evaluation, monitoring, isolation, memory, and model/framework agnosticism

The talk describes production deployment of three major agent products and a platform direction intended to scale from hundreds of agents to thousands.