AI SRE Agents for Production System Diagnostics

cleric · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production incident diagnostics

Cleric is building AI SRE agents that can diagnose production incidents, but production environments are dynamic, sparse in labels, hard to reproduce, and risky enough that bad agent noise quickly destroys operator trust.

Constraints:

• production graphs and infrastructure context go stale quickly
• enterprise systems have bespoke tools, tribal knowledge, and custom access patterns
• agents must not create alert fatigue or spam engineers with low-confidence findings
• the system initially has read-only production access
• real incidents provide sparse and incomplete labels, so customer data alone cannot fully train the agent

Architecture

Cleric’s AI SRE uses a continuously updated production knowledge graph, tool abstractions over logs/metrics/traces/cloud systems, memory layers for customer-specific procedures and episodes, a confidence score plus critiquer to decide whether to notify humans, and an external evaluation bench that mirrors production tools. Background scanning updates the graph within budget limits, investigation agents use the graph and tools to root-cause alerts or engineer-submitted issues, and human feedback on findings becomes sparse positive or negative training signal.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  production-environment["production-environment"] --> background-graph-scanner["background-graph-scanner"]
  production-environment["production-environment"] --> tool-abstraction-layer["tool-abstraction-layer"]
  production-environment["production-environment"] --> diagnostic-agent["diagnostic-agent"]
  background-graph-scanner["background-graph-scanner"] --> production-knowledge-graph["production-knowledge-graph"]
  efficient-scanning-models["efficient-scanning-models"] --> background-graph-scanner["background-graph-scanner"]
  production-knowledge-graph["production-knowledge-graph"] --> diagnostic-agent["diagnostic-agent"]
  tool-abstraction-layer["tool-abstraction-layer"] --> diagnostic-agent["diagnostic-agent"]
  memory-layer["memory-layer"] --> diagnostic-agent["diagnostic-agent"]
  diagnostic-agent["diagnostic-agent"] --> confidence-critiquer["confidence-critiquer"]
  confidence-critiquer["confidence-critiquer"] --> human-sre["human-sre"]
  human-sre["human-sre"] --> feedback-signal-store["feedback-signal-store"]
  feedback-signal-store["feedback-signal-store"] --> memory-layer["memory-layer"]
  tool-abstraction-layer["tool-abstraction-layer"] --> evaluation-bench["evaluation-bench"]
  evaluation-bench["evaluation-bench"] --> memory-layer["memory-layer"]

Components

Data flow

Background jobs inspect infrastructure, Kubernetes objects, deployments, Slack/runbook context, and other production signals to update a graph. When an alert fires or an engineer asks for help, the diagnostic agent traverses that graph, calls observability and production tools, consults customer-specific memories and procedures, and searches for likely root causes. A confidence score and critiquer decide whether the agent should respond, stay quiet, ask for more interaction, or hand back to a human. Findings, engineer reactions, accepted recommendations, and handcrafted eval-bench labels feed future improvement.

Why this architecture

Cleric uses graph context and tools because first-principles exploration of a live production system is too slow and expensive. The confidence gate protects engineer attention because noisy findings would get the product removed. Tool abstraction makes eval scenarios transferable to production, while customer-specific memory sits on top of a generalized base agent so improvements can transfer across customers without losing local procedures.

Models used: AI SRE diagnostic agent · knowledge graph · confidence score · critiquer · evaluation bench · efficient background models

Scale

• The transcript cites a common production-issue prior that 50-60% of issues may be tied to config or code changes.
  “50 or 60% of their production issues were just due to config or code changes”transcript, stated
• Investigation budgets can be capped per investigation, such as 10 cents or one dollar.
  “per investigation let's say make it 10 cents or make it a dollar or whatever”transcript, stated
• The speaker uses 3-7 minute agent runs as a proxy for likely failure and about 30 seconds as a proxy for likely correctness.
  “has the agent run for 3 4 5 6 7 minutes it's probably wrong”transcript, stated

  “came to a conclusion quickly like in 30 seconds it's probably going to be right”transcript, stated
• The agent initially has read-only production access.
  “we only have read in the AIS so our agent cannot make change”transcript, stated
• The production-reproduction problem is described as virtually impossible at most companies at scale.
  “doing this at scale is virtually impossible at most companies”transcript, stated

Alternatives considered (roads not taken)

Investigate production problems only from first principles

Rejected because: The graph lets the agent search much more efficiently than blindly trying tools and hypotheses.

Continuously scan all logs and cloud data with expensive reasoning models

Rejected because: The team treats background scanning like a budgeted lay-of-the-land pass and uses more efficient models because of volume.

Always notify engineers when the agent finds something

Rejected because: Noisy low-confidence findings would worsen alert fatigue and destroy trust.

Train only from real customer feedback

Rejected because: Production feedback is sparse and lopsided, so a handcrafted eval bench is needed for coverage.

Patterns

agent-memory (state)

Cleric stores customer-specific procedures, memories, past episodes, and engineering feedback so the agent can reason with local operational context.

tool-use-orchestration (agent-architecture)

The diagnostic agent depends on tools for logs, metrics, traces, configuration, cloud state, and eval-world access rather than raw model reasoning alone.

confidence-thresholding (safety)

The agent uses confidence scoring and a critiquer to decide when to surface findings and when to stay quiet.

human-in-the-loop-gate (safety)

Humans can authorize further investigation, take over at budget limits, and provide explicit feedback on bad or useful findings.

cost-aware-routing (optimization)

High-volume background scanning uses cheaper models and budgets, while deeper reasoning is reserved for investigations.

eval-driven-development (evaluation)

A controlled eval bench with handcrafted labels gives coverage that sparse production feedback cannot provide.

observability-tracing (observability)

The speaker expects trace-style high-cardinality event data to be more agent-useful than human-centric metric dashboards.

Tradeoffs

Keep early production access read-only

Chose diagnosis and recommendations over automatic remediation in live systems.

Rationale: Read-only access reduces operational risk while the product earns trust before moving toward closed-loop resolution.

Spend more initially on context visibility

Chose show the agent more information out of the gate over prematurely minimizing context for cost and latency.

Rationale: The speaker says the system typically errs toward being valuable first, then optimizes cost and latency later.

Use a production-like fake world for evals

Chose transferable tool abstractions and simulated scenarios over eval tools and worlds that differ from production.

Rationale: Keeping tools and worlds similar improves transfer when the agent moves from eval scenarios to real customer environments.

Gotchas (what bit them)

Production knowledge graphs become stale almost immediately.

Root cause: Production systems change quickly through deployments, IP changes, pod renames, and configuration updates.

Resolution: Continuously update the graph through background scanning and investigation-time discoveries rather than treating it as static documentation.

Agent failures can consume budget slowly while producing little value.

Root cause: Agents may chase recursive loops or weak directions when they do not have a strong path to root cause.

Resolution: Use run-time duration, confidence scoring, and a critiquer as failure signals before notifying humans.

Human-centric observability dashboards are not necessarily agent-friendly.

Root cause: LLMs struggle with metrics and time series compared with semantic sources like code, config, and logs.

Resolution: Use tools and data abstractions better suited to agents, with trace/high-cardinality event data as a likely better substrate.

Customer-specific production context does not transfer cleanly from one environment to another.

Root cause: Each company has bespoke systems, shorthand, procedures, and access patterns.

Resolution: Layer localized customer procedures and memories on top of a generalized base agent and eval bench.

Evaluation

Cleric combines sparse production feedback with a controlled eval bench. Production feedback comes from human reactions, accepted recommendations, system health after changes, generated PRs, and implicit interaction signals; the external eval bench provides handcrafted labels and tool-transferable scenarios for broader coverage.

Metrics: confidence score · agent run duration such as 30 seconds versus 3-7 minutes · human positive/negative feedback on findings · post-change system health · accuracy on handcrafted eval-bench labels · budget spent per investigation

Outcomes

• Cleric built a diagnostic AI SRE that helps teams root-cause alerts and engineer-submitted issues
• knowledge graphs and background scanning improve root-cause efficiency
• confidence scoring reduces low-value notifications to humans
• eval benches and production feedback together improve the agent despite sparse labels

The transcript is a product/architecture discussion rather than a deployment case with customer counts; concrete scale claims center on per-investigation budgets, 3-7 minute failure-duration heuristics, and the difficulty of reproducing enterprise production systems at scale.