Building Stateful Learning Agents for Production SRE

cleric · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: site reliability engineering agents

Cleric argues that stateless SRE agents cannot become reliable production coworkers because they start fresh each session, fail to retain corrections, and cannot accumulate knowledge about incidents, infrastructure, teams, or procedures.

Constraints:

• Production SRE work depends on environment-specific incident history, observability context, team procedures, and prior corrections.
• Wrong answers are inevitable, so the system must make correction easy and ensure mistakes are not visibly repeated.
• A team-based agent must propagate one user's correction into future sessions for other users.
• Ambient learning can absorb useful context but can also compound stale documentation, dead code, or incorrect history.
• The talk gives production lessons and deployment scale, but not low-level retrieval or storage implementation details.

Architecture

Cleric's learning SRE agent connects to infrastructure and observability systems, sits in alert channels, assists incident investigations, proposes persistent memories, self-harvests reusable skills, captures user ratings, and applies operational memory across sessions. The architecture is organized around a learning loop: act, learn through correction and ambient context, visibly apply knowledge, and let users correct bad knowledge.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  infrastructure-integrations["infrastructure integrations"] --> learning-sre-agent["learning SRE agent"]
  observability-integrations["observability integrations"] --> learning-sre-agent
  alert-channel["alert channel"] --> learning-sre-agent
  alert-channel --> team-users["team users"]
  learning-sre-agent --> persistent-memories["persistent memories"]
  learning-sre-agent --> self-harvested-skills["self-harvested skills"]
  learning-sre-agent --> user-rating-signals["user rating signals"]
  learning-sre-agent --> knowledge-visibility-layer["knowledge visibility layer"]
  persistent-memories --> knowledge-visibility-layer
  self-harvested-skills --> knowledge-visibility-layer
  knowledge-visibility-layer --> team-users

Components

Data flow

Infrastructure and observability integrations feed environment context into the Cleric agent; alert-channel history and live incident discussions provide ambient team context; investigations and user corrections create persistent memories, self-harvested skills, and ratings; later investigations retrieve and display learned knowledge so users can see, correct, or benefit from it.

Why this architecture

Cleric's design follows the claim that agent capabilities are becoming commoditized and learning is the differentiator: the agent must accumulate environment knowledge, team knowledge, and past outcomes while closing a visible correction loop so users keep teaching it.

Scale

• Cleric reports deployments to dozens of customers.
  “We've deployed it to dozens of customers at this point”transcript, stated
• The lessons are based on months and nearly years of trial and error.
  “what we've learned over months and to you know now bordering on years of trial and error”transcript, stated
• The correction loop is explicitly designed for team-based use where corrections must affect later sessions.
  “It's especially important for a team-based agent like Clerk where one user session and the the corrections that are learned from that apply to the next user session.”transcript, stated

Alternatives considered (roads not taken)

A stateless agent that starts each session fresh.

Rejected because: The talk defines learning agents as the opposite of that model and argues stateless coworkers lack memory of past work.

Correction without visible improvement.

Rejected because: Cleric says users lose trust when they teach the agent something and the lesson is not retained.

Only learn from explicit user correction.

Rejected because: The presenter says relying only on user correction limits learning to what users explicitly direct.

Ambient learning without user correction.

Rejected because: Cleric warns that ambient learning without correction can compound errors from dead code or outdated documentation.

Fine-tuning or adjusting model weights.

Rejected because: In Q&A, the speaker says Cleric is not doing fine-tuning and believes context engineering is possible today.

Patterns

agent-memory (agent architecture)

The agent persists memories and procedures across investigations so later sessions can reuse corrections and learned context.

data-quality-loop (learning loop)

Corrections, ratings, and visible application of learned knowledge create a feedback loop for improving the agent.

observability-tracing (operations)

The SRE agent connects to observability tooling and alert channels to build operational context for incident investigation.

human-in-the-loop-gate (feedback)

Users teach the agent by correcting mistakes and seeing those corrections influence future work.

Tradeoffs

Build a learning agent instead of a stateless one.

Chose Persistent operational memory over Fresh-session stateless behavior.

Rationale: SRE usefulness depends on accumulated environment, team, and incident knowledge rather than isolated actions.

Use context engineering rather than fine-tuning.

Chose Context engineering over Fine-tuning or weight adjustment.

Rationale: The speaker says Cleric is not fine-tuning and expects improving foundation models to make context-based learning easier.

Learn ambiently from real work while preserving correction controls.

Chose Ambient context absorption plus user correction over Only explicit teaching or unchecked ambient learning.

Rationale: The talk argues explicit-only learning is too narrow, but ambient learning without correction compounds errors.

Gotchas (what bit them)

A correction loop fails if users cannot see improvement.

Root cause: Users lose trust when they spend effort teaching an agent and the lesson is not retained.

Resolution: Make correction persistence, compounding, and visibility part of the product behavior.

Ambient learning can compound bad operational knowledge.

Root cause: Production environments include dead code, outdated documentation, and stale procedures that an agent may absorb as truth.

Resolution: Pair ambient learning with user correction and visible knowledge application.

Explicit-only learning bottlenecks the agent.

Root cause: If the agent only learns when users directly correct it, learning coverage is limited to user-directed moments.

Resolution: Place the agent in real workflows so it can absorb environment-specific context automatically.

Evaluation

The talk presents qualitative production learning from deployments to dozens of customers over months to nearly years, rather than a formal quantitative benchmark.

Metrics: Deployed to dozens of customers · Months to bordering on years of trial and error · Three stated learning-agent lessons · No formal incident-resolution metric disclosed

Outcomes

• Cleric reports deploying the learning SRE agent to dozens of customers.
• The team crystallized three production lessons: easy correction, rewarded/visible correction, and continuous context absorption.
• The talk frames operational memory as the missing piece that completes the learning loop for agents.

Deployment scale is described as dozens of customers and months to nearly years of production trial and error; no incident-volume, latency, or accuracy metrics are disclosed.