AI-Powered SRE Agent for Production Infrastructure Management

cleric_ai · Tech · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production infrastructure operations

Cleric is building an AI SRE teammate that can investigate production alerts across complex distributed infrastructure, reduce engineer context switching, and move incident response from reactive dashboard hunting toward diagnosis, remediation, and eventually prevention.

Constraints:

• modern Kubernetes and microservice graphs are too complex for engineers to hold in working memory
• scripts, runbooks, and more dashboards do not solve unique incidents that require judgment
• the agent must work inside customer production environments and existing observability tools
• engineers will ignore the agent if it produces noisy, low-confidence, or overwhelming findings
• full automatic remediation is still guarded behind human action

Architecture

Cleric deploys an AI SRE agent into the customer's VPC, where it connects to existing observability, logs, metrics, knowledge bases, and production tools rather than replacing them. Alerts trigger a reasoning engine that plans, executes tool calls, reflects, and branches across multiple root-cause hypotheses. A knowledge graph models relationships among teams, clusters, VMs, services, and deployments, while a learning/memory layer stores successful investigations and human feedback. Confidence scoring decides whether to present concise Slack-context findings, expose deeper investigation traces, rerun with feedback, propose solutions, or remain silent.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  infrastructure-event-trigger["infrastructure-event-trigger"] --> reasoning-engine["reasoning-engine"]
  reasoning-engine["reasoning-engine"] --> tool-access-layer["tool-access-layer"]
  reasoning-engine["reasoning-engine"] --> knowledge-graph["knowledge-graph"]
  reasoning-engine["reasoning-engine"] --> confidence-scorer["confidence-scorer"]
  reasoning-engine["reasoning-engine"] --> human-slack-interface["human-slack-interface"]
  tool-access-layer["tool-access-layer"] --> production-environment["production-environment"]
  production-environment["production-environment"] --> tool-access-layer["tool-access-layer"]
  production-environment["production-environment"] --> knowledge-graph["knowledge-graph"]
  knowledge-graph["knowledge-graph"] --> memory-bank["memory-bank"]
  memory-bank["memory-bank"] --> confidence-scorer["confidence-scorer"]
  memory-bank["memory-bank"] --> human-slack-interface["human-slack-interface"]
  confidence-scorer["confidence-scorer"] --> human-slack-interface["human-slack-interface"]
  human-slack-interface["human-slack-interface"] --> remediation-gate["remediation-gate"]

Components

Data flow

An infrastructure event or alert automatically triggers Cleric. The agent generates competing hypotheses such as auth-service, database, or deployment causes, then branches into investigative paths. It calls existing observability and infrastructure tools, combines their results with pre-existing knowledge-graph context, and iterates until it reaches a diagnosis. It tracks confidence during the investigation, presents concise findings to humans when confidence is high enough, and can expose deeper reasoning, tools called, memories, or rerun-with-feedback actions for verification and future learning.

Why this architecture

Cleric's architecture mirrors how SREs debug: start from an alert, form hypotheses, inspect related systems, correlate metrics/logs/traces, and present a concise diagnosis. Deploying in the VPC and using existing tools avoids forcing customers to replace observability stacks. The knowledge graph reduces search space across distributed infrastructure, while confidence scoring and progressive disclosure protect engineers from noisy agent output. Human-gated remediation reflects the team's current trust boundary.

Models used: LLMs · customer-provided models · knowledge graph · memory store

Scale

• Cleric was live with enterprise customers at the time of the talk.
  “it's live it's are you live with claric we are live at enterpris here”transcript, stated
• Even a small customer Kubernetes cluster produced an expensive relationship graph of pods, deployments, and other resources.
  “it's actually a very small cluster but just the amount of connections here is very expensive”transcript, stated
• The speaker uses 500 Datadogs and 500 service graphs as the scale of dashboard overload engineers face.
  “if you throw 500 data dogs at this problem you'll still be left with more dashboards more information but you need action”transcript, stated

  “with like 500 Services uh with 500 CPU graphs and memory graphs and latency graphs”transcript, stated
• A Cleric investigation can touch hundreds or thousands of systems and return in seconds or minutes.
  “touching hundreds or thousands of systems and within seconds or sometimes minutes it gives you a very concise answer”transcript, stated
• Past successful memories are used when the system has solved a problem 50 times.
  “we' we've solved this problem 50 times we're pretty confident we can do it again”transcript, stated

Alternatives considered (roads not taken)

Hiring more engineers

Rejected because: More engineers also create more production systems and do not remove the underlying complexity.

Scripts and runbooks

Rejected because: Unique incidents still require human judgment and cannot all be pre-coded.

Adding more logs, metrics, dashboards, and observability tools

Rejected because: More observability creates more information but not the action or diagnosis engineers need.

Automatic remediation today

Rejected because: Cleric currently guards resolution behind human action while focusing first on reliable diagnosis.

Patterns

tool-use-orchestration (agent architecture)

Cleric investigates incidents by calling existing production observability, logs, metrics, APIs, CLIs, and dashboards as tools inside a planning/reflection loop.

agent-memory (learning)

A knowledge graph, memories, and successful past runs help Cleric narrow investigations and gain confidence when similar alerts recur.

confidence-thresholding (human interface)

Cleric decides when to speak based on confidence, suppressing uncertain findings to avoid wasting engineer attention.

human-in-the-loop-gate (operations)

Diagnoses and proposed solutions are presented to engineers, while remediation remains human-gated rather than fully autonomous.

Tradeoffs

Use existing observability and tools instead of replacing them.

Chose An agent operator over current tools over A new observability platform.

Rationale: Cleric is meant to operate like an engineer in the team, using DataDog-like systems as tools rather than duplicating them.

Suppress low-confidence findings.

Chose Silence or concise high-confidence messages over Dumping raw findings to engineers.

Rationale: If the agent overwhelms engineers, they ignore it and stop providing feedback, weakening the learning loop.

Focus first on diagnosis before remediation and prevention.

Chose Accurate plans, findings, and diagnoses over Immediate closed-loop autonomous fixes.

Rationale: The team wants to establish reliable confidence by problem class before allowing fixes or preventative actions.

Gotchas (what bit them)

Trust and product adoption were harder than LLM mechanics.

Root cause: Introducing an agent as an actual teammate changes team workflows and attention patterns.

Resolution: Use concise high-confidence outputs, progressive disclosure, feedback actions, and local control services so engineers can teach and steer the agent.

Metrics are harder for agents than logs or documents.

Root cause: Metric-based debugging requires causal comparison across many service graphs rather than simple text search.

Resolution: Build specialized metric tools and an agent-computer-interface layer to reduce the search space.

Naive LLM confidence scoring is not grounded enough for production operations.

Root cause: Blindly asking an LLM to score every alert ignores whether the system has actually solved similar cases before.

Resolution: Classify and label incoming alerts, retrieve memories of successful past runs, and compute confidence from those dimensions.

Infrastructure debugging lacks clean external verifiers.

Root cause: Unlike software tests, production infrastructure failures are distributed, stateful, and hard to restage.

Resolution: Source ground truth from multiple operational signals and keep the domain modular rather than assuming one verifier.

Evaluation

Cleric measures agent performance across planning, findings, diagnosis, resolution, confidence, user feedback, and offline evaluation benches, while using high-confidence recurrence over labeled memories as an operational trust signal.

Metrics: plan alignment · finding accuracy · diagnosis accuracy · resolution accuracy · confidence score quality · human feedback through rerun-with-feedback and proposed-solution actions

Outcomes

• Cleric was live with enterprise customers and positioned as an AI SRE teammate rather than a replacement for observability tools.
• The system could automatically trigger from infrastructure events, investigate many paths concurrently, and present concise diagnoses to humans.
• The roadmap was diagnosis first, then class-specific remediation, then preventative actions.

The talk gives qualitative and example scale: enterprise deployments, small Kubernetes clusters with expensive relationship graphs, 500-service dashboard examples, hundreds or thousands of systems touched in seconds or minutes, and 50 prior memories for confidence.