Building Self-Learning AI Agents for Site Reliability Engineering, Visual Asset Review, and Software Development

cleric_/puntt/_tanagram · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production agent reliability across SRE, visual compliance, and software development

Cleric, Puntt, and Tanagram each needed to make agentic systems reliable in production: Cleric for incident response trust, Puntt for high-accuracy visual asset review, and Tanagram for industrialized AI-assisted software development.

Constraints:

• Agents lose trust when they take almost every correct step but make one high-impact final recommendation.
• Private domain knowledge differentiates production agents more than public model knowledge.
• Human feedback can be wrong, so learned rules must be scoped and verified.
• Vision-language models alone cannot deliver pixel-perfect marketing compliance review.
• Enterprise visual review requires high accuracy because regulatory mistakes can trigger expensive recalls.
• Coding agents need fast reliable foundations, self-verification, eval replay, cloud skills, and shareable transcripts to operate at high volume.

Architecture

The talk presents three production agent architectures. Cleric's SRE agent learns a company's private operational domain through initial infrastructure mapping, expert feedback, and prior investigations treated as hypotheses. Puntt's review system uses deterministic computer-vision tools (OCR, layout, color, coordinates) to build a high-confidence asset representation, reconciles tool outputs with an LLM, then lets a review agent identify guideline violations and produce pinned designer feedback. Tanagram's software factory treats agent-assisted development as a production system: fast linters/type checks/tests run in parallel, agents perform end-to-end self-verification, eval tooling supports replay and self-improvement, prompts run as cloud skills, and agent-session threads are captured for PR context and collaboration.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  cleric-alert-intake[Cleric alert intake] --> cleric-infrastructure-map[Cleric infrastructure map]
  cleric-alert-intake --> cleric-sre-agent[Cleric SRE agent]
  cleric-infrastructure-map --> cleric-sre-agent
  cleric-sre-agent --> expert-feedback-memory[Expert feedback memory]
  cleric-sre-agent --> prior-investigation-memory[Prior investigation memory]
  cleric-sre-agent --> verification-step[Verification step]
  expert-feedback-memory --> verification-step
  prior-investigation-memory --> verification-step
  puntt-asset-intake[Puntt asset intake] --> ocr-extractor[OCR extractor]
  puntt-asset-intake --> layout-analyzer[Layout analyzer]
  puntt-asset-intake --> color-analyzer[Color analyzer]
  puntt-asset-intake --> coordinate-extractor[Coordinate extractor]
  ocr-extractor --> reconciliation-llm[Reconciliation LLM]
  layout-analyzer --> reconciliation-llm
  color-analyzer --> reconciliation-llm
  coordinate-extractor --> reconciliation-llm
  reconciliation-llm --> review-agent[Review agent]
  review-agent --> pinned-designer-output[Pinned designer output]
  coordinate-extractor --> pinned-designer-output
  tanagram-foundation-harness[Tanagram foundation harness] --> tanagram-coding-agents[Tanagram coding agents]
  tanagram-foundation-harness --> posthog-metrics[PostHog metrics]
  tanagram-coding-agents --> self-verification-loop[Self verification loop]
  tanagram-coding-agents --> eval-self-improvement-loop[Eval self improvement loop]
  tanagram-coding-agents --> cloud-skills[Cloud skills]
  tanagram-coding-agents --> thread-archive[Thread archive]

Components

Data flow

Cleric starts from an alert, uses a prebuilt infrastructure map to narrow the tool/search space, consults scoped expert feedback and prior investigations, then verifies whether a remembered hypothesis applies before acting. Puntt ingests a visual asset, runs OCR, layout analysis, color analysis, and coordinate extraction in parallel, reconciles those outputs with an LLM into a high-confidence representation, then sends that representation to a review agent that applies guidelines and emits pinned visual feedback. Tanagram runs agent work against fast foundations, asks agents to verify real end-to-end behavior, uses eval replay and self-improvement to improve code, deploys prompt-based cloud skills, and stores agent threads as PR and collaboration context.

Why this architecture

All three systems reduce the parts of the task left to raw LLM intuition. Cleric learns private operational context but verifies it instead of freezing it into rigid rules. Puntt gives the LLM deterministic visual evidence because compliance needs precision that visual-language models do not provide alone. Tanagram builds fast deterministic foundations and real self-verification because coding agents run repeatedly and must prove actual behavior, not just produce plausible code.

Models used: Cleric self-learning SRE agent · Vision-language models · Reconciliation LLM · Puntt review agent · Claude / cloud-code style coding agents · BitFab evaluation/self-improvement tooling

Scale

• Cleric had been building its SRE agent for about two and a half years.
  “we've been building this for about I'd say like two and a half years.”transcript, stated
• The sample Cleric alert involved consumer lag spiking to 4,000 over two minutes.
  “there's some consumer lag that spiked to 4,000 right over the last two minutes”transcript, stated
• Without mapping, Cleric might search across 20 systems and make hundreds of tool calls.
  “out of 20 different systems I'm going to figure out what are the right systems for me to look at”transcript, stated

  “search every single system. Where is this event processor? Now that's like hundreds of tool calls, lots of context”transcript, stated
• Cleric uses time-to-resolution comparisons such as 8 minutes without memory, 90 seconds with good memory, and 16 minutes with bad memory.
  “if a run with no memory if I took eight minutes to do that with memory did I take 90 seconds if I take 16 minutes then I probably learned a whole bunch of garbage”transcript, stated
• Puntt's interview task uses 10 to 20 assets to test visual review approaches.
  “we have something like a set of guidelines and then a set of like 10 to 20 assets that we need to find all of the problems in.”transcript, stated
• Puntt says enterprise visual compliance needs 95% accuracy.
  “For us, we need accuracy to be at 95%.”transcript, stated
• Puntt cites potential $10 million recall cost for a packaging compliance error.
  “if I send it out to the printers, it might cost me $10 million in a recall.”transcript, stated
• Tanagram reports about 40 pull requests per day with five engineers.
  “ship you know 40 PRs a day at our team while maintaining um high quality with a with a team of five engineers.”transcript, stated
• Tanagram's local foundation harness finishes lint, typecheck, and tests within 15 seconds.
  “run all three checks locally in parallel and they all finish within 15 seconds.”transcript, stated
• A Tanagram optimization reduced lint from 10 to 5 seconds, typecheck from 13 to 9.6 seconds, and tests from 15 to 13.8 seconds.
  “lint used to take 10 seconds got down to 5 seconds the type check used to take 13 got down to 9.6 and then our test suite went from 15 to 13.8”transcript, stated

Alternatives considered (roads not taken)

Compete on public knowledge

Rejected because: Foundation model labs will absorb public knowledge into future model weights, eroding differentiation.

Treat engineer feedback as absolute truth

Rejected because: Humans can give wrong feedback and fail to correct it later, causing the agent to learn false rules.

Reuse prior incidents as facts

Rejected because: A past root cause may not apply to today's alert; historical memory should seed hypotheses, not decisions.

Feed images and rules directly to a vision-language model

Rejected because: Marketing compliance requires precise color, layout, measurements, text, and coordinates that VLMs handle inconsistently.

Mock agent verification flows

Rejected because: Mocking may not test the real auto-update or upload behavior; agents can verify by operating across the actual stack.

Patterns

agent-memory (learning)

Cleric stores private operational knowledge from mappings, expert feedback, and prior investigations, but scopes and verifies remembered rules.

tool-use-orchestration (architecture)

Cleric narrows infrastructure tools through mapping, and Puntt orchestrates OCR, layout, color, coordinate, and LLM components rather than relying on one model.

eval-driven-development (quality)

Cleric uses time-to-resolution as a memory-quality metric, Puntt works against a 95% accuracy requirement, and Tanagram treats evals as replay plus self-improvement.

data-quality-loop (data)

Puntt first builds the most precise asset interpretation possible with deterministic tools, then uses an LLM to reconcile and review that evidence.

observability-tracing (operations)

Tanagram monitors foundational check latency in PostHog and captures agent threads for PR context and debugging.

batch-offline-processing (workflow)

Puntt reviews thousands of visual assets by running specialized preprocessing and review over assets before designers act on pinned outputs.

Tradeoffs

Learn private domain knowledge

Chose Company-specific maps, expert feedback, and past investigations over Relying on public model knowledge.

Rationale: Foundation labs will absorb public knowledge, but team-specific context remains the durable differentiator for production agents.

Use prior work as hypotheses

Chose Verify remembered causes at incident time over Treat historical investigations as facts.

Rationale: History accelerates investigation only when the agent checks that the same condition is actually present today.

Hybrid CV plus LLM asset review

Chose Deterministic visual tools feeding an LLM reconciliation/review layer over One vision-language model doing all interpretation and review.

Rationale: Traditional tools provide precision; the LLM reconciles ambiguity and applies guidelines.

Agent self-verification with real systems

Chose End-to-end terminal/API/CI/npm verification over Unit mocks for every failure mode.

Rationale: Agents can operate the actual systems under test, which catches behavior mocks may miss.

Capture agent threads as durable collaboration artifacts

Chose Thread archive attached to PRs and forkable by teammates over Treating agent sessions as disposable chats.

Rationale: Threads explain why code was built a certain way and let teammates continue research conversations.

Gotchas (what bit them)

One wrong recommendation can erase trust even after many correct investigation steps.

Root cause: The agent lacked private operational knowledge that the Kafka lag alert happened daily due to an analytics cron job.

Resolution: Invest in domain learning from mapping, experts, and prior investigations so the agent understands company-specific context.

Agents chase red herrings when the search space is too broad.

Root cause: Without an infrastructure map, the agent can query every suspicious system and fill context with irrelevant paths.

Resolution: Pre-map the environment and restrict investigations to relevant tool subsets.

User feedback can teach the wrong lesson.

Root cause: Humans sometimes give negative feedback and later realize the agent was right, or never correct the false negative.

Resolution: Scope learned rules narrowly and avoid trusting feedback as absolute truth.

Vision-language models underperform on pixel-level visual compliance.

Root cause: They approximate spatial, graphical, color, and text details that enterprise compliance review needs precisely.

Resolution: Use deterministic OCR, layout, color, and coordinate tools before LLM reconciliation and review.

Slow or flaky foundations compound across agent runs.

Root cause: Coding agents run linters, type checks, tests, and CI paths hundreds of times per day.

Resolution: Optimize and monitor foundations as production infrastructure.

Evaluation

The three systems use different evaluation strategies: Cleric measures whether memory reduces incident time-to-resolution; Puntt targets enterprise-grade visual-review accuracy; Tanagram requires fast foundation checks, replay/self-improvement eval tooling, and end-to-end self-verification rather than mocked slices.

Metrics: Cleric time-to-resolution with and without memory · Puntt visual review accuracy target · Exact coordinates and timestamps for visual issues · Tanagram lint/typecheck/test latency · PostHog alerts for foundation metric breaches · Eval replay and self-improvement · End-to-end self-verification success

Outcomes

• Cleric frames domain learning as the core differentiator for trustworthy self-learning SRE agents.
• Puntt improved visual asset review by combining deterministic CV tools with LLM reconciliation and review.
• Puntt targets 95% accuracy for high-stakes enterprise compliance work.
• Tanagram reports shipping about 40 PRs per day with a five-engineer team.
• Tanagram reduced foundation-check latencies and uses agent threads for PR context and collaboration.

The case spans a two-and-a-half-year SRE-agent effort, thousands of visual assets reviewed for enterprise brands, 95% compliance accuracy requirements, and a software-factory workflow producing about 40 PRs/day with five engineers.