Agentic System for Autonomous Code Monitoring and Maintenance

ramp · Finance · 2026 · source Agentic: yes · promotional bias: med · summary-only (no fetched source) · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Autonomous product maintenance

Ramp wanted to reduce production-monitoring toil and catch quality issues in Ramp Sheets quickly without making engineers own every observability task.

Constraints:

sandboxed reproduction
human review before merge
production alert triage

Architecture

Ramp built a self-maintenance loop for Ramp Sheets on top of Ramp Inspect. The system first tried nightly autonomous QA, then moved to Datadog-triggered monitors generated on pull-request merge. When a monitor fires, a sandboxed Inspect session receives alert context, reproduces the issue against live code, validates a fix, and opens a reviewed pull request. A triage layer filters noisy alerts, tunes/deletes bad monitors, posts real issues to Slack, and stores PR links on monitor objects so later agents do not duplicate work.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  pr-merge-monitor-generator[PR-merge monitor generator] --> ai-generated-datadog-monitors[AI-generated Datadog monitors]
  ai-generated-datadog-monitors --> datadog-webhook[Datadog webhook]
  datadog-webhook --> ramp-inspect-session[Ramp Inspect session]
  ramp-inspect-session --> triage-agent[Triage agent]
  triage-agent --> monitor-state-store[Monitor state store]
  monitor-state-store --> ai-generated-datadog-monitors
  ramp-inspect-session --> pull-request-workflow[Pull request workflow]
  triage-agent --> slack-notification[Slack notification]
  pull-request-workflow --> slack-notification

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`PR-merge monitor generator`	Reads code diffs on merge and creates Datadog monitors for newly introduced behavior.	`AI-generated Datadog monitors`
`AI-generated Datadog monitors`	Granular observability mesh that detects error, latency, and behavior regressions in Ramp Sheets.	`PR-merge monitor generator`, `Datadog webhook`, `Monitor state store`
`Datadog webhook`	Starts an agent session when a monitor fires and passes alert context into the maintenance workflow.	`AI-generated Datadog monitors`, `Ramp Inspect session`
`Ramp Inspect session`	Sandboxed coding-agent environment that can make real API requests, run tests, reproduce bugs, and validate fixes.	`Datadog webhook`, `Triage agent`, `Pull request workflow`
`Triage agent`	Assesses whether an alert is real or noise, tunes/deletes noisy monitors, and suppresses duplicates.	`Ramp Inspect session`, `Monitor state store`, `Slack notification`
`Monitor state store`	Uses monitor descriptions to store PR links and duplicate-suppression state.	`AI-generated Datadog monitors`, `Triage agent`
`Pull request workflow`	Carries proposed fixes to engineer review before merge.	`Ramp Inspect session`, `Slack notification`
`Slack notification`	Alerts the team when the agent finds a real issue or prepares a fix.	`Triage agent`, `Pull request workflow`

Data flow

On code merge, an agent reads the diff and creates granular Datadog monitors for the new behavior. When a monitor detects a regression, a webhook starts a Ramp Inspect session with the alert context. The agent reproduces the bug in a sandbox, develops and validates a fix, and notifies the team or opens a pull request. A triage pass first decides whether the alert is real; for duplicates it reads state stored on the monitor description and stands down.

4 supporting quotes

“First, on PR merge, an agent reads the code diff and generates monitors instrumenting the new code.”summary, stated

“Second, when a monitor fires, a Datadog webhook triggers a new agent session with the alert context. The agent then reproduces the issue in its sandbox environment, develops and validates a fix, and notifies the team on Slack.”summary, stated

“To address this, the team added a triage step where on every alert, the agent first assesses the scope and validity of the problem.”summary, stated

“When an agent pushes a fix, it appends the PR link to the monitor description. Subsequent agents that encounter the same fired monitor see the link and stand down, avoiding redundant work.”summary, stated

Why this architecture

Ramp moved away from broad nightly QA because unfocused agents repeatedly explored the same paths and missed narrow situational bugs. Traditional observability provided the focusing signal, while sandboxed reproduction made agent fixes less speculative by requiring the agent to observe and validate the failure before sending humans a pull request.

3 supporting quotes

“Without a specific mission or triggering context, the agent would progress down similar paths in its QA workflow each night, reading the same files, investigating the same features, and running the same tests.”summary, stated

“The sandboxed reproduction capability represents a key architectural decision that distinguishes this system from simpler log-analysis or static-code-review approaches.”summary, stated

“This suggests the focused, monitor-driven approach was adopted not by choice but by necessity due to model limitations.”summary, implied

Models used: GPT-5 models · Claude Opus 4.6

Scale

Ramp Sheets monitoring grew from ten handwritten monitors to more than a thousand generated monitors.
“The system scaled Ramp Sheets from ten hand-written monitors to over a thousand in just a few weeks.”summary, stated
The monitoring density reached roughly one monitor per 75 lines of code.
“The one-monitor-per-75-lines-of-code metric is striking and represents an unprecedented level of observability granularity.”summary, stated
The first week produced 40 real bug catches within minutes of user-triggered issues.
“In its first week of operation, the system caught 40 real bugs within minutes of occurrence, significantly reducing the observability burden on engineering teams while improving product quality and reducing downtime for customers.”summary, stated

Alternatives considered (roads not taken)

Nightly scheduled autonomous QA

Rejected because: It found broad issues but repeatedly explored similar paths and lacked trigger context for narrow bugs.

“However, this nightly agent approach had significant limitations. Without a specific mission or triggering context, the agent would progress down similar paths in its QA workflow each night”summary, stated

Notify humans for every generated-monitor alert

Rejected because: Poor thresholds caused cascades of false positives and duplicate notifications, so alert triage became mandatory.

“Auto-generated monitors had poorly tuned thresholds, causing routine user activity to trigger cascades of alerts, most being false positives.”summary, stated

Replace all trusted hand-written monitors with agent-generated ones

Rejected because: Agent monitors are powerful but opaque, so critical issues still need instrumentation engineers wrote and trust.

“Agent-generated monitors are powerful but opaque and not yet reliable enough to be the only line of defense. When critical issues occur, teams still want instrumentation they wrote and trust.”summary, stated

Patterns

`observability-tracing` (operations)

The system creates dense AI-powered monitors and uses alerts as the starting point for autonomous investigation.

“The scale of deployment is notable: the system operates using approximately one thousand AI-generated monitors covering every 75 lines of code in the codebase.”summary, stated

`human-in-the-loop-gate` (control)

The agent can propose fixes but code changes still wait for engineer review before merging.

“The system continuously monitors production code, triages alerts, and generates fixes without human intervention, though all code changes still require engineer review before merging.”summary, stated

Tradeoffs

How to scale observability work.

Chose agent-generated monitoring and repairs over manual observability ownership by engineers.

Rationale: Ramp traded more autonomous infrastructure for reduced operational burden while preserving review gates.

“and easily parallelizable labor that makes exhaustive monitoring feasible at production scale; teams generally dislike owning observability work, and offloading this to agents allows faster shipping; and strong observability combined with automated QA leads to fewer bugs, less downtime, and bette”summary, implied

Triggering strategy for maintenance agents.

Chose monitor-driven activation with alert context over nightly general QA sweeps.

Rationale: Focused alerts gave agents a specific mission and exposed narrow production bugs that broad scheduled review missed.

“The second iteration pivoted to using Datadog monitors to direct agents at specific production issues.”summary, stated

Human attention management.

Chose agent triage and duplicate suppression before Slack notifications over posting every fired generated monitor to humans.

Rationale: Generated monitors were granular but noisy; triage preserves trust by tuning bad monitors and suppressing duplicate work.

“If it's a real issue, the agent pushes a fix and posts to Slack. If it's noise, the agent tunes or deletes the monitor.”summary, stated

Observability trust boundary.

Chose keep traditional hand-written monitors alongside generated monitors over relying solely on opaque agent-generated instrumentation.

Rationale: The team wanted trusted instrumentation for critical incidents while generated monitors matured.

“Keep existing instrumentation: The team recommends maintaining traditional hand-written monitors alongside auto-generated ones.”summary, stated

Gotchas (what bit them)

Static code inspection was not enough for subtle product failures.

Root cause: Many bugs only emerge when the agent reproduces behavior against live code and APIs.

Resolution: Run each maintenance attempt in a sandbox where the agent can make real API requests and execute tests.

“This interactivity is positioned as critical because subtle failure modes are rarely apparent from static code review alone.”summary, stated

Evaluation

Ramp measured real production bugs caught in the first week.

Metrics: 40 real bugs · minutes to detection · monitor density

“In its first week of operation, the system caught 40 real bugs within minutes of occurrence, significantly reducing the observability burden on engineering teams while improving product quality and reducing downtime for customers.”summary, stated

Outcomes

The first week reportedly caught 40 real bugs and reduced observability burden.

Summary-only extraction because the social source was not usable in the local cache.

“The system evolved from nightly scheduled QA agents to a monitor-driven maintenance approach that generates over a thousand AI-powered monitors (one per 75 lines of code) that automatically detect issues, reproduce bugs in sandboxed environments, and create pull requests with fixes.”summary, stated