AI SRE System with Continuous Learning for Production Issue Investigation

cleric_ai · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production incident investigation

Cleric needed its AI SRE agent to learn from live production incidents and compare multiple investigation strategies, even though production states disappear after resolution and customer-specific learnings may not safely generalize.

Constraints:

• production issues are stateful, dynamic, and cannot be reproduced later
• the agent examines multiple systems and strategies concurrently
• investigation feedback must be tied to the exact trace and strategy
• customer-specific details must be stripped before shared learnings are generalized
• generalized memories need measurable impact before being trusted across deployments

Architecture

Cleric’s AI SRE starts investigations automatically when alerts fire, using existing observability tools and read-only access to inspect database metrics, network traffic, logs, and system resources. LangSmith traces parallel investigation paths, compares strategies on the same incident, aggregates performance metrics, and ties Slack/ticket feedback directly to the traces. A continuous-learning layer stores investigation details and resolution patterns, anonymizes customer-specific data, promotes suitable patterns into generalized memories, and measures whether those memories improve future investigation success rates and resolution times.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  production-alert-source["production-alert-source"] --> cleric-investigation-agent["cleric-investigation-agent"]
  cleric-investigation-agent["cleric-investigation-agent"] --> observability-tool-stack["observability-tool-stack"]
  cleric-investigation-agent["cleric-investigation-agent"] --> langsmith-tracing["langsmith-tracing"]
  cleric-investigation-agent["cleric-investigation-agent"] --> engineer-feedback-channel["engineer-feedback-channel"]
  langsmith-tracing["langsmith-tracing"] --> strategy-comparison-dashboard["strategy-comparison-dashboard"]
  engineer-feedback-channel["engineer-feedback-channel"] --> feedback-api["feedback-api"]
  feedback-api["feedback-api"] --> langsmith-tracing["langsmith-tracing"]
  feedback-api["feedback-api"] --> customer-specific-knowledge-space["customer-specific-knowledge-space"]
  customer-specific-knowledge-space["customer-specific-knowledge-space"] --> privacy-anonymization-layer["privacy-anonymization-layer"]
  privacy-anonymization-layer["privacy-anonymization-layer"] --> generalized-memory-library["generalized-memory-library"]
  generalized-memory-library["generalized-memory-library"] --> cleric-investigation-agent["cleric-investigation-agent"]
  generalized-memory-library["generalized-memory-library"] --> strategy-comparison-dashboard["strategy-comparison-dashboard"]

Components

Data flow

When an alert fires, Cleric begins parallel investigations across observability sources using different strategies. Each path is traced in LangSmith with the systems inspected, the strategy used, metrics, and engineer feedback. After the investigation, Cleric stores both specific investigation details and abstracted resolution patterns. Privacy controls remove customer-specific information before generalized memories are made selectively available in future deployments, where LangSmith tracks whether those memories improve success and resolution metrics.

Why this architecture

The system needs tracing because production incidents are ephemeral and multiple strategies must be compared while the incident still exists. LangSmith becomes the observability layer for agent behavior: it links traces, strategies, metrics, and human feedback. Separate local and generalized knowledge spaces are necessary because some operational learnings depend on customer-specific tooling or procedures, while others can benefit all deployments after anonymization and validation.

Models used: Cleric AI SRE agent · LangSmith tracing · LangSmith feedback API · observability tools

Scale

• Cleric analyzes thousands of concurrent traces.
  “analyze investigation patterns across thousands of concurrent traces”article, stated
• The agent examines multiple systems simultaneously during an investigation.
  “Cleric examines multiple systems simultaneously - checking database metrics, network traffic, application logs, and system resources”article, stated
• Learning scope is compared across teams, companies, and industries.
  “By comparing performance metrics across different teams, companies, and industries”article, stated
• The agent works through existing observability stacks rather than requiring new tools for teams to learn.
  “There's no new tooling to learn – Cleric works with existing observability stacks”article, stated

Alternatives considered (roads not taken)

Rely on one successful investigation strategy as generally valid

Rejected because: The team needed data-driven validation because an approach that worked once might not generalize.

Generalize all customer learnings directly

Rejected because: Some solutions depend on internal tools or processes absent in other environments, and privacy controls must strip sensitive details first.

Force teams into a new incident tooling surface

Rejected because: Cleric communicates through Slack and existing observability stacks so engineers can use normal workflows.

Patterns

observability-tracing (observability)

LangSmith traces concurrent investigation paths so the team can compare strategies and tie feedback to the exact trace.

agent-memory (state)

Investigation details and resolution patterns become customer-specific memories or anonymized generalized memories for future investigations.

eval-driven-development (evaluation)

Different investigation strategies are compared side-by-side and measured against success rates, resolution times, and feedback.

human-in-the-loop-gate (safety)

Engineer feedback through Slack and ticketing systems shapes which findings and memories are useful.

privacy-preserving-processing (privacy)

Customer-specific details are anonymized before patterns are analyzed or shared across deployments.

Tradeoffs

Run multiple investigation strategies concurrently

Chose parallel strategy comparison over one fixed investigation path per incident.

Rationale: Concurrent investigations reveal which approaches work best for different issue types while the incident state still exists.

Separate local and generalized knowledge spaces

Chose customer-specific context plus generalized patterns over one shared global memory for all operational learnings.

Rationale: Some memories depend on local procedures, while anonymized generalized memories can benefit all users.

Gotchas (what bit them)

Production incidents are one-shot learning opportunities.

Root cause: After an issue is resolved, the exact stateful environment disappears.

Resolution: Trace concurrent investigations as they happen so the team can learn before the opportunity is gone.

Parallel investigation strategies are hard to compare without trace-level observability.

Root cause: Multiple investigation paths inspect different systems and may run simultaneously on the same incident.

Resolution: Use LangSmith to compare strategies side-by-side, aggregate metrics, and tie user feedback to traces.

A useful customer-specific memory can be unsafe or irrelevant elsewhere.

Root cause: The memory may rely on proprietary data, identifying details, or internal processes unavailable in other deployments.

Resolution: Strip customer details and determine whether the learning should stay local or become generalized.

Evaluation

Cleric evaluates investigation strategies by tracing concurrent paths, comparing approaches on the same incident, tying engineer feedback to traces, and measuring success rates, resolution times, and the impact of generalized memories before and after introduction.

Metrics: investigation success rates · resolution times · performance metrics for investigation approaches · user feedback tied to strategy · usage and impact of shared memories · thousands of concurrent traces

Outcomes

• Cleric gained visibility into parallel production investigations
• feedback can be tied directly to investigation strategy traces
• successful resolution patterns can be turned into local or generalized memories
• shared learnings can be measured for impact across deployments

The article reports analysis across thousands of concurrent traces and describes cross-team, cross-company, and cross-industry comparison for deciding whether memories should remain local or generalize.