Building a Self-Healing Software Factory with AI Agents

software_factory · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Autonomous software development and production bug remediation

Software Factory wanted to build Memo through AI agents that could plan features, implement code, detect production bugs, and repair issues with minimal human handoff, while keeping enough inspection and review control for a real development workflow.

Constraints:

The project was a 10-day build with the system halfway through on day five
Bugs arrived through informal Slack feedback, GitHub issues, and production monitoring
Agents needed enough product specification detail to avoid technically correct but undesired fixes
Production changes have different risk levels, especially database and access-control changes
Self-grading can inflate quality if agents grade their own work without scrutiny
Automated remediation only works when monitoring surfaces the relevant signals

Architecture

The factory combines scheduled and event-triggered Ona agents around a pull-request workflow. A feature planner runs every 30 minutes to triage GitHub issues and backlog items, feature builders and bug fixers implement work, a Slack integration lets team members tag Ona on feedback threads, Sentry feeds production incidents into an incident responder, and a quality.md file lets agents compare implementation quality against product specifications. PR review and risk-sensitive manual gates remain the control surface.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  slack-feedback-channel[Slack Feedback Channel] --> ona-agent-environment[Ona Agent Environment]
  slack-feedback-channel[Slack Feedback Channel] --> bug-fixer[Bug Fixer]
  github-issues[GitHub Issues] --> feature-planner[Feature Planner]
  feature-planner[Feature Planner] --> feature-builder[Feature Builder]
  feature-planner[Feature Planner] --> quality-file[Quality File]
  feature-builder[Feature Builder] --> quality-file[Quality File]
  feature-builder[Feature Builder] --> pull-request-review[Pull Request Review]
  bug-fixer[Bug Fixer] --> pull-request-review[Pull Request Review]
  sentry[Sentry] --> incident-responder[Incident Responder]
  incident-responder[Incident Responder] --> github-issues[GitHub Issues]
  quality-file[Quality File] --> automation-auditor[Automation Auditor]
  pull-request-review[Pull Request Review] --> human-reviewer[Human Reviewer]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Slack Feedback Channel`	Low-friction internal bug-reporting entry point where team members tag Ona.	`Ona Agent Environment`, `Bug Fixer`
`Ona Agent Environment`	Ephemeral agent workspace spun up to investigate Slack-reported bugs.	`Slack Feedback Channel`, `Bug Fixer`
`GitHub Issues`	Structured external and open-source bug/feature request queue.	`Feature Planner`
`Feature Planner`	Scheduled triage agent that enriches issues, assigns priority, and routes work.	`GitHub Issues`, `Feature Builder`, `Quality File`
`Feature Builder`	Implementation agent for planned feature and bug-fix work.	`Feature Planner`, `Quality File`, `Pull Request Review`
`Bug Fixer`	Agent that investigates reported bugs and proposes code fixes.	`Slack Feedback Channel`, `Pull Request Review`
`Sentry`	Production monitoring signal source for runtime errors.	`Incident Responder`
`Incident Responder`	Automation that queries Sentry, deduplicates issues, investigates failures, and creates bug reports.	`Sentry`, `GitHub Issues`
`Quality File`	quality.md benchmark and self-improvement ledger for product-spec coverage and grades.	`Feature Planner`, `Feature Builder`, `Automation Auditor`
`Automation Auditor`	Planned meta-agent for reviewing and improving the factory's automations.	`Quality File`
`Pull Request Review`	Review and approval stage before code merges.	`Feature Builder`, `Bug Fixer`, `Human Reviewer`
`Human Reviewer`	Risk gate for higher-risk production changes and automation validation.	`Pull Request Review`

Data flow

A bug can enter through Slack, GitHub, or Sentry. Slack reports spin up an Ona environment that reads the thread and screenshots, while GitHub issues are picked up by the scheduled feature planner for triage and enrichment. Sentry issues are read by an incident responder automation, which checks whether a bug already exists, investigates the stack trace or database state, files issues, and routes work to builder/fixer agents. Changes then move through PR review, with humans expected for higher-risk production changes.

4 supporting quotes

“Ona spun up an environment to to address this bug. It's going through the Slack channel, reads out the the description, reads out the uh attached screenshots”transcript, stated

“This will be picked up by our feature planner. So, we've discussed quite in detail uh how our automations work. Um our feature planner currently works on a 30-minute schedule”transcript, stated

“Um as you can see, the agent then goes to work, um figures out what's wrong, and then opens up uh issues. Uh and this is uh representative of one of the issues that we have uh we have fixed. You can then go back into GitHub, see the logs of whatever we fixed. One of the examples”transcript, stated

“decide what you want to auto-approve and what you want to just uh yeah, have your eyes over uh or your team look over.”transcript, stated

Why this architecture

The factory uses existing development tools rather than a separate AI workspace because bugs already appear in Slack, GitHub, and Sentry. Scheduled planning, event-triggered bug fixing, quality-file grading, and PR review create a loop that can shorten time-to-action while preserving auditability and selective human oversight.

3 supporting quotes

“Right now like this is almost skipped, and people directly tag So our head of product engineering, Matt, I think his morning routine is like going through the feedback product channel”transcript, stated

“the quality MD file was part of the harness uh initially. And it will be um updated by the agents as we go along”transcript, stated

“it shortens that time to action so much that usually a user wouldn't run into the same bug. I did because I like on the app immediately immediately when it was available. Um but before you could fix my bug, Sentry already locked it, and then um”transcript, stated

Models used: Ona agents · LLM-powered feature planner · LLM-powered feature builder · LLM-powered bug fixer · LLM-powered incident responder

Scale

By day five, the project had 14 automations, 11 running regularly, and 2,000 agent executions.
“and I think there were 14 automations set up by you, 11 are running um um regularly, and 2,000 agent executions have run. And 98% were were automated, so like truly running in the background like not triggered by you manually, but just like you know firing on a on an event or on a schedule.”transcript, stated
Ninety-eight percent of the 2,000 executions were automated.
“And 98% were were automated, so like truly running in the background like not triggered by you manually, but just like you know firing on a on an event or on a schedule. And 2% were of those 2,000 agent executions were manually triggered, which is probably like you”transcript, stated
Two percent of the 2,000 executions were manually triggered.
“schedule. And 2% were of those 2,000 agent executions were manually triggered, which is probably like you setting up the harness and like you know iterating on the system. So it is almost like a fully automated factory line already, which is great.”transcript, stated
The feature planner runs every 30 minutes.
“uh how our automations work. Um our feature planner currently works on a 30-minute schedule, running every 30 minutes. You can also uh trigger it manually. So, we will double-check if this has been picked up in a few minutes already. Otherwise, we'll just trigger this.”transcript, stated

Alternatives considered (roads not taken)

Route Slack feedback into Linear and wait for manual implementation

Rejected because: The team wanted reports to go directly into agent investigation and fixing instead of languishing as handoff work.

“it has been like a place where like lots of product feedback has been shared, right? And then it's like usually has been like very tedious like bringing this into Linear”transcript, stated

“solved it. Right now like this is almost skipped, and people directly tag So our head of product engineering, Matt, I think his morning routine is like going through the feedback product channel and just be like at Ona, at Ona, fix this, fix this, and it's like fires off like a”transcript, stated

Auto-approve every production change indefinitely

Rejected because: Database and access-control changes should require manual review in stricter production environments.

“we currently run everything on auto-approve. In a production environment, you would probably do something that changes the database access uh with a manual review.”transcript, stated

Enable the automation auditor immediately

Rejected because: The factory needed a stable base system and human validation before letting meta-automation improve automation loops.

“the automation auditor uh I think yesterday very briefly. The idea for the automation auditor is really to be our improvement loop”transcript, stated

“wait for the automation loop and the automation auditor loop to come into place because we see that the automations are not perfect out of the box”transcript, stated

Patterns

`multi-agent-orchestration` (agent architecture)

The factory separates feature planning, building, bug fixing, incident response, and automation auditing into specialized agents.

“it's mainly the uh the feature builder and the feature planner now as well as the bug fixer updating this uh this file.”transcript, stated

“what the um incident responder um um automation does is like it shortens that time to action so much”transcript, stated

`observability-tracing` (operations)

Sentry issues feed an incident responder that investigates production errors and opens issues for repair.

“Sentry is our monitoring uh loop for the actual app. And there are some bugs that uh were found directly with uh Sentry.”transcript, stated

`eval-driven-development` (quality)

The quality.md file decomposes the product spec, grades architecture areas, and identifies gaps for agent improvement.

“we decompose, for example, the individual grades and occasionally, whenever the backlog is empty, look if there's any gaps”transcript, stated

`human-in-the-loop-gate` (operations)

The factory can distinguish auto-approved low-risk changes from higher-risk database changes that need human review.

“decide what you want to auto-approve and what you want to just uh yeah, have your eyes over uh or your team look over.”transcript, stated

`async-message-orchestration` (operations)

Slack threads, GitHub issues, scheduled planners, Sentry events, and PR reviews form asynchronous work queues for agents.

“you can have an automation just run on it um whenever there is a new bug being opened and really have this autonomous flow”transcript, stated

“This will be picked up by our feature planner.”transcript, stated

Tradeoffs

Use Slack tagging for internal bugs

Chose Low-friction Ona mentions in feedback threads over Manual transfer from Slack to issue trackers before implementation.

Rationale: The direct Slack path reduced handoff friction and let a product lead fire off many bug-fixing agents quickly.

“our head of product engineering, Matt, I think his morning routine is like going through the feedback product channel and just be like at Ona, at Ona, fix this”transcript, stated

Use GitHub issues for external/open-source reports

Chose Structured public issue intake over Slack-only bug reporting.

Rationale: Open-source users would not be in the internal Slack, and GitHub issues give a structured entry point agents can triage.

“this is going to be open sourced, right? And then probably after this um week we're going to open up the issue list for for everyone to add issues.”transcript, stated

Keep human gates for high-risk changes

Chose Manual review for database access changes in production over Universal auto-approval.

Rationale: The team wanted velocity but recognized that risk appetite varies by change type and production environment.

“In a production environment, you would probably do something that changes the database access uh with a manual review.”transcript, stated

Gotchas (what bit them)

Vague specifications led to technically correct but wrong fixes.

Root cause: Agents optimized for the reported symptom without enough guidance on the intended solution approach.

Resolution: The team emphasized more detailed product specs and more specific examples for quality grades.

“our product spec was detailed but not detailed enough. And this leads ultimately to the agent coming up with an outcome that is suboptimal.”transcript, stated

“the more accurate and specific you are with your examples, how you can get to an actual grade, the better your outcome will be.”transcript, stated

Self-grading risked inflated A grades.

Root cause: The same agent loop that built features also updated grades, leaving room for loose interpretation.

Resolution: The team planned more scrutiny and more detailed success criteria around grading.

“you have to scrutinize the self-grading as well.”transcript, stated

“And we probably want to be a little bit more uh scrutinizing of giving like an A even if we know like it is probably not like an A grade”transcript, stated

Production blind spots limited automated healing.

Root cause: If errors are swallowed or monitoring queries miss warnings, the automations cannot see the relevant failures.

Resolution: The team treated observability coverage as part of the factory and improved Sentry querying.

“if you, you know, have a setup that swallows some of the errors, um you will obviously not catch them”transcript, stated

“the automation uh should do this itself, and the auditor should catch this.”transcript, stated

Evaluation

The project measured automation volume, automated/manual execution share, PR throughput, quality.md grades, Sentry-detected incidents, and production bug remediation.

Metrics: 14 automations · 11 running regularly · 2,000 agent executions · 98% automated executions · 2% manual executions · 30-minute feature planner cadence · letter-grade quality.md scoring

3 supporting quotes

“there were 14 automations set up by you, 11 are running um um regularly, and 2,000 agent executions have run.”transcript, stated

“And 98% were were automated, so like truly running in the background like not triggered by you manually”transcript, stated

“this ultimately translates to, you know, in this case, uh four or five letter scale.”transcript, stated

Outcomes

The day-five factory had 14 automations and 2,000 agent executions
Ninety-eight percent of executions were automated
Sentry-fed incident response fixed a workspace creation failure caused by a row-level security rule
Slack bug reports could directly spawn agent environments and fixes
The team identified more rigorous grading and observability coverage as next improvements

The source describes a five-day snapshot of a 10-day Memo build with 14 automations, 11 active automations, and 2,000 agent executions.

3 supporting quotes

“there were 14 automations set up by you, 11 are running um um regularly, and 2,000 agent executions have run.”transcript, stated

“because there was a a row-level security rule preventing new inserts. So, this was flagged to Sentry.”transcript, stated

“the way you respond to it is is different here”transcript, stated

Building a Self-Healing Software Factory with AI Agents

Problem

Architecture

Components

Data flow

Why this architecture

Scale

Alternatives considered (roads not taken)

Route Slack feedback into Linear and wait for manual implementation

Auto-approve every production change indefinitely

Enable the automation auditor immediately

Patterns

multi-agent-orchestration (agent architecture)

observability-tracing (operations)

eval-driven-development (quality)

human-in-the-loop-gate (operations)

async-message-orchestration (operations)

Tradeoffs

Use Slack tagging for internal bugs

Use GitHub issues for external/open-source reports

Keep human gates for high-risk changes

Gotchas (what bit them)

Vague specifications led to technically correct but wrong fixes.

Self-grading risked inflated A grades.

Production blind spots limited automated healing.

Evaluation

Outcomes

`multi-agent-orchestration` (agent architecture)

`observability-tracing` (operations)

`eval-driven-development` (quality)

`human-in-the-loop-gate` (operations)

`async-message-orchestration` (operations)