Building an Autonomous AI SRE Agent for Production Incident Investigation

datadog · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Autonomous SRE incident investigation

Datadog needed Bits AI SRE to investigate production incidents across noisy telemetry and multi-component distributed systems, producing root-cause analyses rather than shallow summaries.

Constraints:

Architecture

Bits AI SRE mimics human SRE investigation: it forms root-cause hypotheses, validates or rejects them using targeted telemetry queries, follows supported evidence into deeper sub-hypotheses, prunes unsupported branches, and repeats until it reaches a root cause. Datadog evaluates the agent on a benchmark dataset of real labeled incidents and monitor alerts, using archived telemetry and LLM judges aligned to human judgments.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  incident-alert[Incident Alert] --> bits-ai-sre[Bits AI SRE]
  bits-ai-sre --> hypothesis-engine[Hypothesis Engine]
  hypothesis-engine --> telemetry-query-tools[Telemetry Query Tools]
  telemetry-query-tools --> production-telemetry[Production Telemetry]
  hypothesis-engine --> bits-ai-sre[Bits AI SRE]
  bits-ai-sre --> root-cause-report[Root Cause Report]
  benchmark-harness[Benchmark Harness] --> bits-ai-sre[Bits AI SRE]
  production-telemetry --> benchmark-harness[Benchmark Harness]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Incident Alert Monitor alert or incident needing investigation. Bits AI SRE
Bits AI SRE Autonomous SRE agent that reasons over telemetry and produces RCA. Hypothesis Engine, Telemetry Query Tools, Root Cause Report, Benchmark Harness
Hypothesis Engine Creates, validates, rejects, and recursively expands hypotheses. Bits AI SRE, Telemetry Query Tools
Telemetry Query Tools Targeted access to logs, traces, metrics, and other telemetry. Hypothesis Engine, Production Telemetry
Production Telemetry Live or archived observability data for incidents. Telemetry Query Tools, Benchmark Harness
Benchmark Harness Real-incident evaluation dataset with labels and LLM/human-aligned scoring. Bits AI SRE, Production Telemetry
Root Cause Report Audit-ready investigation output for engineers. Bits AI SRE

Data flow

An incident or monitor alert starts an investigation. Bits AI SRE gathers context, forms hypotheses, runs targeted telemetry queries, validates or rejects each branch, recursively explores promising sub-hypotheses, and produces an audit-ready root-cause analysis. Benchmark evaluation feeds archived incident telemetry and scores conclusions against ground truth.

“by forming hypotheses, testing them using live telemetry data, and following promising evidence to a root cause.”article, stated
“We worked across hundreds of teams at Datadog to collect and label real incidents and used them to create a benchmark dataset of test scenarios. We evaluate Bits AI SRE’s performance by scoring its output against each scenario’s ground truth label.”article, stated

Why this architecture

Datadog moved away from broad telemetry summarization because tool-call fanout created distracting noise and context-window pressure. Hypothesis-driven targeted queries keep the agent focused on causal relationships and enable deeper multi-component investigations.

“Bits AI SRE investigates like a team of SREs performing an on-call investigation. Rather than simply summarizing raw telemetry data all at once, Bits AI SRE investigates like a human.”article, stated
“Increasing the number of tool calls caused the input token count for the summarization prompt to scale linearly.”article, stated

Models used: Bits AI SRE agent · LLM judge

Scale

Alternatives considered (roads not taken)

Summarize many telemetry tool responses at once.

Rejected because: Noise and linear token growth led to wrong root causes and context-limit pressure.

“But because other tool responses included suspicious signals like critical application errors in an upstream service, the summarization prompt returned an incorrect root cause.”article, stated

Stop at superficial alert cause.

Rejected because: Complex incidents require deeper causal chains, such as payload size causing OOMs.

“While this answer is superficially correct, the newest version of Bits AI SRE digs one level deeper to surface that the OOMs were caused by an influx of abnormally large payloads, which led a single pod to crash, triggering the alert.”article, stated

Evaluate on synthetic incidents only.

Rejected because: Real incident data is critical for measuring generalization to production complexity.

“Evaluating against real incidents is fundamental and critical to building an effective AI SRE agent. This is the most reliable way to measure meaningful progress and ensure that an agent can generalize to the complexity of real-world environments.”article, stated

Patterns

tool-use-orchestration (sre-agent)

Bits AI SRE runs targeted telemetry queries as part of hypothesis testing.

“Validate or reject hypotheses using data from targeted queries”article, stated

eval-driven-development (incident-evaluation)

Datadog regularly measures agent performance against real labeled incidents.

“We use this benchmark to regularly measure our agent’s performance and improve the agent over time.”article, stated

llm-as-judge (evaluation)

The evaluation workflow uses an LLM judge aligned with human judgments.

“an LLM judge scores the agent’s conclusions across multiple criteria.”article, stated

Tradeoffs

Investigation strategy

Chose Hypothesis-driven causal search over Broad summarization of telemetry.

Rationale: It reduces noise and follows evidence toward root cause.

“following the evidence where it leads.”article, stated

Evaluation basis

Chose Real labeled incidents over Toy or synthetic examples.

Rationale: Real telemetry captures production complexity and generalization demands.

“real-world environments”article, stated

Gotchas (what bit them)

More telemetry queries can make the answer worse.

Root cause: Irrelevant suspicious signals distract a summarization prompt and grow the context.

Resolution: Use targeted causal hypothesis testing instead of summarizing everything.

“other tool responses included suspicious signals like critical application errors in an upstream service”article, stated

Superficially correct root causes can miss the deeper operational cause.

Root cause: A pod running out of memory was a symptom, not the underlying abnormal-payload cause.

Resolution: Recursively generate deeper hypotheses until the search space is exhausted.

“recursively generates deeper root cause hypotheses until it exhausts the search space”article, stated

Evaluation

Datadog evaluates Bits AI SRE on a benchmark of real labeled incidents and monitor alerts, using archived telemetry and LLM judge scoring aligned with human responder judgments.

Metrics: Pass/fail against ground truth · Time to resolution · Benchmark score · Root-cause correctness · Causal depth · Human-aligned judge scores

“incidents and monitor alerts are labeled by human responders, archived telemetry is fed to the Bits AI SRE agent”article, stated

Outcomes

The source reports hundreds of labeled Datadog team incidents, 12 tool calls in an early failure example, root-cause analyses in minutes, and up-to-95% time-to-resolution reduction.

“The newest version of Bits AI SRE correctly surfaces the commit latency as the root cause”article, stated