Building a Software Factory with AI Agents and Automation Loops

software_factory · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI-assisted software development

The Memo team wanted a software factory where agents could plan, implement, review, repair, and monitor a product without humans manually driving each pull request.

Constraints:

• New feature work still needed human intent and issue definition, while fix and review loops could be automated.
• Pull requests had to satisfy review conversations and rule-set requirements before merging.
• The system needed fallbacks for failed automations, stale pull requests, merge conflicts, and production bugs.
• Compliance-aware review meant low-risk changes could be accelerated, but humans still needed accountability for final merge decisions.

Architecture

Memo's software factory centers GitHub issues and pull requests as the durable work queue. Feature-planning and feature-building automations create and implement work, a PR reviewer opens blocking GitHub conversations, agent loops resolve those conversations, a PR shepherd revisits stale or needs-human PRs, and post-merge/incident automations connect CI and Sentry back into bug reports and improvement loops.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  github-issues[GitHub Issues] --> feature-planner[Feature Planner]
  feature-planner --> feature-builder[Feature Builder]
  feature-builder --> pull-request[Pull Request]
  pull-request --> pr-reviewer[PR Reviewer]
  pr-reviewer --> agent-resolution-loop[Agent Resolution Loop]
  agent-resolution-loop --> pull-request
  pull-request --> pr-shepherd[PR Shepherd]
  pr-shepherd --> agent-resolution-loop
  agent-resolution-loop --> automation-auditor[Automation Auditor]
  sentry[Sentry] --> incident-responder[Incident Responder]
  incident-responder --> github-issues
  pull-request --> post-merge-verification[Post Merge Verification]
  post-merge-verification --> github-issues

Components

Data flow

A user or human creates a GitHub issue, the feature planner decides whether to request more information or hand it to the feature builder, the builder opens a pull request, the PR reviewer creates comments that must be resolved, automated loops revise the code until conversations close, and fail-safe/post-merge automations pick up stale, failed, or production-error cases.

Why this architecture

The architecture treats every development bottleneck as a feedback loop: agents do the routine implementation/review/fix work, humans move up to designing loops and reviewing higher-level artifacts, and redundant automations catch failures when primary loops stall.

Scale

• The day-four demo had already merged 52 pull requests.
  “We looked in the into the 52 PRs that were merged already. We're now at a stage where the initial outline and the initial product spec is actually live. So you can go into and we'll actually have a look in”transcript, stated
• The feature builder ran hourly.
  “We run it hourly. But you can also trigger it manually if you wanted to, but obviously manual work defeats the purpose here. So once the feature builder picks up, creates a push”transcript, stated
• The PR shepherd fail-safe ran about once an hour.
  “do? I like the sound of that. So PR shepherd is basically a a fail-safe mode that runs currently on a schedule about once an hour. Um the idea behind it is that for PRs like the other one that have hit like a needs human situation, the shepherd actually reviews”transcript, stated
• The incident responder polled Sentry every 15 minutes.
  “all the bugs that we uh, capture through Sentry go through the incident responder. This runs on a schedule every 15 minutes”transcript, stated
• The review architecture was designed for environments that might raise hundreds or thousands of PRs per day.
  “if you're trying to build a software factory and you're raising, you know, hundreds if not thousands of pull requests a day, that just becomes completely untenable.”transcript, stated

Alternatives considered (roads not taken)

Manual GitHub issue and PR handling

Rejected because: Manual work defeated the point of a software factory and could not keep up with autonomous development loops.

Escalating every needs-human pull request directly to a person

Rejected because: The team added a PR shepherd that first checks whether the original needs-human assessment was wrong and fixes agent-resolvable issues.

Line-by-line human review of all agent-generated diffs

Rejected because: The workflow moved human review toward architecture diagrams, entity diagrams, and bottom-line impact artifacts.

Full compliance-style multi-reviewer approval for every generated PR

Rejected because: The team described risk-classified auto-approval for low-risk changes so review effort stays focused on complex changes.

Patterns

multi-agent-orchestration (agent architecture)

Separate automations handle planning, building, PR review, comment resolution, stale-PR shepherding, auditing, and incident response.

durable-execution (orchestration)

GitHub issues, PR conversations, labels, schedules, and stale-state checks preserve work across repeated autonomous runs.

fallback-degradation (resilience)

The PR shepherd acts as a secondary loop when the primary review and resolution flow stalls or misclassifies an issue as requiring a human.

observability-tracing (operations)

Daily metrics, CI performance checks, and Sentry-connected incident response feed operational signals back into development work.

human-in-the-loop-gate (governance)

The system reduces human involvement for low-risk changes while preserving human merge responsibility and escalation paths.

Tradeoffs

Development-loop ownership

Chose Humans designing and supervising feedback loops over Humans manually implementing each feature or fix.

Rationale: The team wanted humans to operate at the loop/architecture level while agents performed repeated implementation, review, and fix cycles.

Risk-based approval model

Chose Auto-approval for low-risk code paths with human merge accountability over Mandatory full human review for every PR.

Rationale: The approach preserves compliance and accountability while avoiding a review bottleneck for simple, non-critical changes.

Redundant review loops

Chose Secondary PR shepherd review over Trusting the first automation's needs-human/stale state.

Rationale: A second loop can catch automation failures, stale branches, and wrongly escalated PRs before they stop the factory.

Meta-automation for factory improvement

Chose Automation auditor reviewing automations and runs over Manual periodic audit of duplicated or inefficient automation steps.

Rationale: The same loop-based approach can identify redundant review work and suggest improvements to the factory itself.

Gotchas (what bit them)

Autonomous PRs can hit a wall and mark themselves as needing a human even when the underlying issue is agent-fixable.

Root cause: The primary automation can fail, stall, or misclassify limitations without a secondary reviewer.

Resolution: Add a PR shepherd fail-safe that rechecks stale or needs-human PRs on a schedule and fixes what agents can fix.

Review becomes the bottleneck once agents can raise large volumes of PRs.

Root cause: Compliance-style multi-reviewer controls do not scale to hundreds or thousands of agent-generated pull requests.

Resolution: Classify low-risk changes for auto-approval while reserving human attention for complex or sensitive changes.

Human reviewers cannot efficiently inspect every generated diff at software-factory scale.

Root cause: Agent-generated changes produce too many low-level code artifacts for traditional review to remain the primary oversight mechanism.

Resolution: Shift review toward bottom-line summaries, architecture diagrams, domain-boundary checks, and other higher-level artifacts.

Evaluation

The factory evaluates PRs through automated code review, blocking comment-resolution conversations, post-merge verification, CI performance monitoring, daily metrics, and Sentry-backed incident response.

Metrics: Merged pull requests · Lines of code written · Test coverage · CI performance checks · Sentry-captured client/server bugs · Stale or needs-human PRs

Outcomes

• The system had already merged 52 PRs by the day-four demo.
• PR review, comment resolution, stale-PR recovery, daily metrics, and incident-response loops were operating or being activated.
• Humans shifted toward supervising loops, architecture artifacts, and risk decisions instead of direct GitHub work.

The demo describes a live factory for Memo with hourly builder/shepherd loops and a 15-minute Sentry incident-responder loop; some auditor and performance automations were still upcoming.