AI Agent for Automated Root Cause Analysis in Production Systems

cleric · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production incident root-cause analysis

Cleric needed agents to investigate production alerts across observability stacks and return actionable root causes without excessive human digging.

Constraints:

ground truth scarcity
user feedback limits
diverse infra environments
high accuracy expectations

Architecture

Cleric’s RCA agent runs as an asynchronous production-alert investigator. PagerDuty or Slack alerts trigger an agent that inspects production systems, observability data, logs, metrics, code, Slack chatter, dependency graphs, and APIs. It plans and executes diagnostic tasks, gathers sparse evidence into dense contextual findings, and returns root-cause candidates or action-ready artifacts for engineers. The evaluation layer records traces of tool calls/reasoning, post-processes them into task clusters and heat maps, and tries to reduce reliance on human feedback because production incidents lack clean ground truth.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  alert-trigger[Alert trigger] --> rca-planning-agent[RCA planning agent]
  rca-planning-agent --> observability-connectors[Observability connectors]
  observability-connectors --> production-evidence-sources[Production evidence sources]
  production-evidence-sources --> finding-synthesizer[Finding synthesizer]
  rca-planning-agent --> finding-synthesizer
  finding-synthesizer --> engineer-review-surface[Engineer review surface]
  rca-planning-agent --> trace-evaluator[Trace evaluator]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Alert trigger`	PagerDuty or Slack production alert that starts investigation.	`RCA planning agent`
`RCA planning agent`	Plans diagnostic tasks, calls APIs, reasons over evidence, and decides when to continue or stop.	`Alert trigger`, `Observability connectors`, `Finding synthesizer`, `Trace evaluator`
`Observability connectors`	Accesses metrics, logs, traces, code context, dependency graphs, Slack chatter, and production APIs.	`RCA planning agent`, `Production evidence sources`
`Production evidence sources`	Sparse operational data spread across logs, time series, code, communications, and service dependencies.	`Observability connectors`, `Finding synthesizer`
`Finding synthesizer`	Condenses retrieved evidence into contextual findings or root-cause candidates for engineers.	`RCA planning agent`, `Production evidence sources`, `Engineer review surface`
`Engineer review surface`	Shows findings, metrics, logs, and dependency graphs without requiring engineers to understand the agent internals.	`Finding synthesizer`
`Trace evaluator`	Records tool calls/reasoning and clusters failures into heat maps for agent improvement.	`RCA planning agent`

Data flow

An alert fires in PagerDuty or Slack. The agent plans an investigation, calls observability and production APIs, searches logs, metrics, code, Slack discussions, and dependency graphs, then condenses those sparse signals into findings and a root-cause hypothesis. Engineers receive artifacts they already understand, while traces of the agent run are post-processed into evaluation heat maps showing repeated task failures such as log analysis or metric querying.

5 supporting quotes

“when an alert fires in your page of duty or Slack, we have an agent that can diagnose that by looking at your production systems, your observability stack, planning, executing tasks, calling APIs”transcript, stated

“there's logs there's code there's chatter in Slack and what you're really trying to do is really first step is information retrieval it's search”transcript, stated

“maybe switching from one tool to the next tool, gathering the data, trying to put two and two together”transcript, stated

“we started with traces. So on the individual run you can see what the agent did like which tools it called what the reasoning was the prompts and all those things.”transcript, stated

“we built like a summarization or like a post-processing process that's completely well it's like part AI part like deterministic but it it collapses and condenses and clusters all of these things together”transcript, stated

Why this architecture

The workflow is asynchronous and artifact-oriented because interrupting engineers in synchronous Slack/helpdesk conversations was hard and high-friction. RCA alerts are better suited to an ambient background agent that performs the slow tool-switching and data gathering, then tees up findings for domain experts. The evaluation design exists because production alerts lack reliable ground truth and human review is itself a bottleneck.

3 supporting quotes

“with the alert flow it's a lot more asynchronous there's not necessar it's a system generating alert you can investigate that on your own time”transcript, stated

“we're also trying to take the grunt work away from engineers that they're not immediately trying to solve. So, it's more like an ambient background agent that's just doing all this work for you.”transcript, stated

“ground truth. Like it's a lack of ground truth in production environment. Unlike code or writing, there's not just this corpus of like web information that you could just download and then train online.”transcript, stated

Models used: LLM agents

Scale

The use case spans heterogeneous production data: time series, logs, code, Slack chatter, metrics, and dependency graphs.
“it's like a lot of information time series there's logs there's code there's chatter in Slack”transcript, stated
The target experience is an ambient agent that can clear many alerts in the background.
“if you check in, you're like, "Well, okay, it's solved like 20 alerts for me. I don't have to go and look at those."”transcript, stated
Evaluation looks for task-level recurring failures rather than only pass/fail incident answers.
“we focus heavily on the tasks that's where most our failures happen. Um let's say one task is analyze the logs you know for in data dog”transcript, stated

Alternatives considered (roads not taken)

Synchronous Slack/helpdesk teammate agent

Rejected because: Getting between engineers involved immediate answers, chitchat, and back-and-forth; alerts were a better asynchronous workload.

“we actually started on the help desk and we're fielding questions from engineers. So, one engineer be like a platform team supporting another engineer coming in with like a question. And getting in between engineers was very hard”transcript, stated

Depend on humans for every review and label

Rejected because: Users may not know if the root cause is real, so human-dependent feedback blocks improvement.

“sometimes the users don't know like if you go into an engineer and say um you know is this the root cause? often times they'll say this looks good but I'm not sure if it's real”transcript, stated

Expose agent internals as the review surface

Rejected because: Cleric wants engineers to assess familiar findings, metrics, logs, and graphs rather than the product internals.

“what they should be assessing is the findings and the metrics and the logs and the dependency graphs and all those things that they already know. Well, we don't want them to have to understand the internals of our product.”transcript, stated

Patterns

`observability-tracing` (operations)

The agent diagnoses alerts by looking through production systems and observability data.

“So when an alert fires in your page of duty or Slack, we have an agent that can diagnose that by looking at your production systems, your observability”transcript, stated

`eval-driven-development` (quality)

Cleric treats ground-truth validation and feedback loops as core challenges for RCA agents.

“So one of them is ground truth.”transcript, implied

Tradeoffs

How much human involvement to require in RCA.

Chose minimize human intervention during investigation over asking engineers to validate every step.

Rationale: Human feedback is useful but can be unavailable or uncertain, so the agent needs independent verification where possible.

“secondary problem effectively the thing that we've learned is you need to as much as possible get the human out of the loop”transcript, stated

Interaction model for incident automation.

Chose asynchronous background investigation of alerts over a synchronous Slack/helpdesk agent between engineers.

Rationale: Alerts can be investigated off the critical human path, while support conversations demanded immediate interactive answers.

“with the alert flow it's a lot more asynchronous there's not necessar it's a system generating alert you can investigate that on your own time”transcript, stated

Evaluation signal.

Chose trace-derived task clustering and machine-checkable metrics over human labels for every RCA outcome.

Rationale: Ground truth is scarce and humans are uncertain, so the system mines traces for recurring tool/task failures that do not always need human feedback.

“many of these metrics you don't need humans for feedback. It's completely like you can just parse the information and then from that we build these heat maps”transcript, stated

Gotchas (what bit them)

It is hard to know whether an RCA agent truly solved the incident.

Root cause: Production incidents lack simple downloadable labels and users may not know the root cause either.

Resolution: Build feedback and verification mechanisms while reducing unnecessary human intervention.

“So you need to figure out whether the agent even success success successfully solved the problem like how do you know that? Um so you need to find ways either user feedback but sometimes the users don't”transcript, stated

Evaluation

The discussion emphasizes ground truth, user feedback, and verification rather than reporting a single benchmark metric.

Metrics: ground-truth availability · feedback quality · root-cause correctness

“The system faces significant challenges around ground truth validation, user feedback loops, and the need to minimize human intervention while maintaining high accuracy across diverse infrastructure environments.”summary, stated

Outcomes

The product aims to turn alert grunt work into distilled findings for engineers.

Transcript discussion, not a published benchmark result.

“Cleric's agent aims to automate this "grunt work" and present engineers with a distilled analysis that leverages their existing knowledge and intuition.”summary, stated