Building a Real-World Evaluation Platform for Autonomous SRE Agents

datadog · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Autonomous SRE agent evaluation

Datadog needed to evaluate Bits AI Investigation across realistic production incidents because changes that looked good on small internal tests could quietly regress unrelated SRE scenarios, and live investigations could not be reliably replayed after telemetry expired.

Constraints:

Architecture

Datadog built a replayable evaluation platform for Bits AI SRE around two core pieces: a curated label set of representative production investigations and an orchestration platform that runs Bits against those labels, scores trajectories, and reports historical performance. The system turns product feedback and incident telemetry into labels, uses Bits itself to assist validation, reconstructs noisy world snapshots before telemetry expires, runs full evaluation sets weekly, and routes results into dashboards, Slack notifications, and product decisions.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  customer-feedback[Customer Feedback] --> label-pipeline[Label Pipeline]
  investigation-telemetry[Investigation Telemetry] --> label-pipeline[Label Pipeline]
  label-pipeline[Label Pipeline] --> agentic-label-validator[Agentic Label Validator]
  agentic-label-validator[Agentic Label Validator] --> bits-investigation[Bits Investigation]
  agentic-label-validator[Agentic Label Validator] --> human-reviewer[Human Reviewer]
  label-pipeline[Label Pipeline] --> label-set[Label Set]
  investigation-telemetry[Investigation Telemetry] --> world-snapshot[World Snapshot]
  world-snapshot[World Snapshot] --> simulated-world[Simulated World]
  label-set[Label Set] --> evaluation-orchestrator[Evaluation Orchestrator]
  evaluation-orchestrator[Evaluation Orchestrator] --> simulated-world[Simulated World]
  simulated-world[Simulated World] --> bits-investigation[Bits Investigation]
  evaluation-orchestrator[Evaluation Orchestrator] --> reporting-infrastructure[Reporting Infrastructure]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Bits Investigation Autonomous SRE agent that investigates production incidents across telemetry and configuration signals. Evaluation Orchestrator, Agentic Label Validator, Simulated World
Customer Feedback Feedback on investigations that seeds label creation. Label Pipeline
Investigation Telemetry Metrics, logs, traces, infrastructure metadata, network telemetry, and monitor context from incidents. Label Pipeline, World Snapshot
Label Pipeline Turns feedback and telemetry into evaluation labels with root causes and world snapshots. Customer Feedback, Investigation Telemetry, Agentic Label Validator, Label Set
Agentic Label Validator Uses Bits to assemble causal chains, resolve ambiguity, and score label confidence before human review. Bits Investigation, Label Pipeline, Human Reviewer
Human Reviewer Validates and refines generated labels when confidence is low or ambiguity remains. Agentic Label Validator, Label Set
Label Set Representative collection of production investigation scenarios with ground truth and snapshots. Label Pipeline, Evaluation Orchestrator
World Snapshot Archived structure and relationships of signals available at incident time. Investigation Telemetry, Simulated World
Simulated World Isolated replay environment that mirrors the original investigation context with realistic noise. World Snapshot, Bits Investigation, Evaluation Orchestrator
Evaluation Orchestrator Runs Bits against labels, scores trajectories, and coordinates evaluation batches. Label Set, Simulated World, Bits Investigation, Reporting Infrastructure
Reporting Infrastructure Dashboards, observability, metadata, Slack notifications, and historical score tracking. Evaluation Orchestrator

Data flow

Customer feedback and investigation telemetry become candidate labels. Bits helps derive causal chains and resolve ambiguous references, low-confidence labels go to human review, and the accepted label stores ground truth plus a world snapshot of available signals. The orchestration platform runs Bits inside isolated simulated worlds, stores scores for each scenario and run, segments results by scenario attributes, and reports regressions through dashboards and Slack.

3 supporting quotes
“When customers provide feedback on a Bits AI investigation, we use that signal, along with the information from the investigation itself, to construct a ground truth root cause analysis”article, stated
“In effect, we build a simulated environment that mirrors the original investigation context, then run Bits inside it. Each environment is fully isolated at the data layer so that investigation context from one label can not affect another.”article, stated
“On the reporting side, we store scores for every scenario across every run. We track these results in Datadog dashboards and Datadog Agent Observability so we can compare performance across agent versions.”article, stated

Why this architecture

Datadog built replayable label-based evaluation because unit-style tool checks and live reruns could not capture multi-step agent failures or changing environments. Preserving noisy world snapshots let Bits face production-like constraints, while embedding label generation into product feedback made coverage scale with usage.

3 supporting quotes
“Bits’ value comes from how it chains tools together and reasons across their outputs. Failures often emerged from interactions between steps, not from a single tool call.”article, stated
“rerunning live Bits investigations as a form of online evaluation. That did not scale. Results were not aggregated, environments changed underneath us”article, stated
“This turned label collection from a manual effort into a pipeline that grows with product usage. As adoption increases, so do the volume and diversity of our labels. Embedding label creation in the product increased our label creation rate by an order of magnitude.”article, stated

Models used: Bits AI · Claude Opus 4.5

Scale

Alternatives considered (roads not taken)

Validate features with small internal test cases

Rejected because: Small tests missed broader regressions across diverse environments.

“Early in development, before exposing the system to customers, we added a feature that extracted the service name from the monitor under investigation into Bits Investigation’s initial context. On the surface, this made sense, and in a handful of internal test cases, it worked as expected.”article, stated
“Without a representative evaluation set, we had no way to measure how that change behaved across different environments.”article, stated

Rerun live investigations as online evaluation

Rejected because: Results were not aggregated, environments changed, and expired signals made replay impossible.

“We also experimented with rerunning live Bits investigations as a form of online evaluation. That did not scale.”article, stated
“We also experimented with rerunning live Bits investigations as a form of online evaluation. That did not scale. Results were not aggregated, environments changed underneath us, and investigations could not be replayed once the underlying signals expired.”article, stated

Snapshot only signals directly tied to root cause

Rejected because: Clean snapshots inflated apparent accuracy by removing realistic noise and red herrings.

“One key discovery was that these simulated worlds need to be noisy. Snapshotting only the signals directly tied to the root cause is not enough. In production, Bits operates in environments full of unrelated services, background errors, and tangential signals.”article, stated
“Without that noise, evaluation results looked better than they should have. We were essentially giving the agent an open-book exam with only the relevant pages. No wonder it aced it. The agent appeared more accurate in these simplified environments than it did in real investigations.”article, stated

Patterns

eval-driven-development (evaluation)

A replayable evaluation platform runs Bits against representative labels to catch regressions and compare agent versions before deployment.

“We needed infrastructure that could turn production issues into reproducible investigation environments. So we built a replayable evaluation platform from scratch.”article, stated

tool-use-orchestration (agent-architecture)

Bits failures come from multi-step tool chains, so evaluation scores the agent's trajectory rather than only the final answer.

“Failures often emerged from interactions between steps, not from a single tool call.”article, stated
“we could evaluate the agent’s trajectory. We looked at how close it got to the correct answer, whether it investigated deeply enough”article, stated

data-quality-loop (evaluation-data)

The platform converts user feedback into labels and improves label quality with confidence scoring, human review, and richer signal reconstruction.

“Every user interaction becomes a potential evaluation label.”article, stated

confidence-thresholding (quality-control)

Generated labels receive confidence scores, and low-confidence outputs are routed to human review.

“each generated label is assigned confidence scores, and anything below a defined threshold is flagged for human review.”article, stated

Tradeoffs

Use noisy world snapshots for evaluation

Chose Broader signal reconstruction with production-like noise over Clean snapshots containing only root-cause signals.

Rationale: Noisy worlds lowered short-term scores but made evaluations more predictive of real production behavior.

“In production, Bits operates in environments full of unrelated services, background errors, and tangential signals.”article, stated
“In the short term, the numbers looked terrible. This reduced our pass rate by roughly 11% and decreased our label count by 35%.”article, stated

Automate label validation while retaining human review

Chose Bits-assisted label generation plus confidence thresholds and human validation over Manual assembly of root cause analyses from raw signals.

Rationale: Manual review could not keep up with ingestion, while agentic validation cut per-label validation time by more than 95%.

“As our label ingestion rate grew, manual review could not keep up.”article, stated
“Validation time per label dropped by more than 95% in a single week”article, stated

Evaluate trajectories instead of only final conclusions

Chose Trajectory-level scoring over Final-answer scoring only.

Rationale: Trajectory scoring revealed whether Bits investigated deeply and surfaced useful telemetry, not only whether it landed on the right root cause.

“Instead of scoring only the final conclusion, we could evaluate the agent’s trajectory.”article, stated
“This allowed us to understand not just whether Bits got the correct answer, but how helpful its investigation was.”article, stated

Gotchas (what bit them)

Small internal tests hid broad agent regressions.

Root cause: The change improved initial context in simple cases but pulled irrelevant signals into unrelated scenarios.

Resolution: Datadog built representative, replayable evaluations before relying on feature-level confidence.

“The feature pulled in a large amount of irrelevant signals, which degraded investigation quality in unrelated scenarios”article, stated
“We needed a way to catch these regressions before users reported them.”article, stated

Live investigation replay did not scale as an evaluation method.

Root cause: Environment state changed and underlying signals expired.

Resolution: The team rebuilt incidents as isolated simulated worlds from captured world snapshots.

“Results were not aggregated, environments changed underneath us, and investigations could not be replayed once the underlying signals expired.”article, stated
“we build a simulated environment that mirrors the original investigation context, then run Bits inside it.”article, stated

Overly clean evaluation worlds overstated agent performance.

Root cause: The first snapshots excluded unrelated services, background errors, and tangential signals that production agents must ignore.

Resolution: Datadog widened signal reconstruction, accepting an 11% pass-rate drop and 35% label-count drop for better predictive validity.

“Snapshotting only the signals directly tied to the root cause is not enough.”article, stated
“This reduced our pass rate by roughly 11% and decreased our label count by 35%. But in the long term, it made our evaluations predictive of production behavior.”article, stated

Evaluation

The platform evaluates Bits on representative labels with ground truth and world snapshots, scores trajectories, segments results by incident dimensions, runs full weekly regression suites, and reports significant deviations.

Metrics: scenario scores across runs · pass@k · root-cause label quality · validation time per label · weekly full-evaluation regression results · significant deviations in overall performance

3 supporting quotes
“The evaluation platform is the system that runs Bits against our label set, scores the results, and tracks performance over time.”article, stated
“metrics like pass@k (for a scenario, given k independent attempts, does the agent succeed on at least one of them).”article, stated
“we run the full evaluation set weekly to catch regressions that may have slipped through.”article, stated

Outcomes

The platform runs Bits weekly against tens of thousands of real-incident scenarios and has expanded to other Datadog agent teams.

3 supporting quotes
“Validation time per label dropped by more than 95% in a single week”article, stated
“We observed roughly a 30% increase in the quality of root causes in the generated labels”article, stated
“Any team building an agent now has access to a large, representative label set and evaluation infrastructure from day one.”article, stated