Building an Autonomous Software Factory for Notion-like Application Development

software_factory · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Autonomous software factory

The Software Factory experiment set out to prove that agents could autonomously build, test, review, deploy, and operate a Notion-like collaborative editing product while also improving the factory itself.

Constraints:

The target product required Notion-like databases, UI, tests, deployment, and operations.
The factory needed a state engine for issues, pull requests, review, and work tracking.
Agent-written code still needed tests, review loops, deployment, and incident response.
Factory improvements sometimes mattered more than application features.

Architecture

The project used GitHub as the factory's state manager and built scheduled automations for planning, feature building, test writing, code review, deployment, operations, and incident response. Agents built Memo, a Notion-like app, while also filing GitHub issues, writing tests where missing, reviewing PRs, deploying fixes, and using Sentry signals to drive automated maintenance.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  github-state-engine[GitHub State Engine] --> builder-agent[Builder Agent]
  builder-agent --> test-runner[Test Runner]
  builder-agent --> github-state-engine[GitHub State Engine]
  github-state-engine --> reviewer-agent[Reviewer Agent]
  reviewer-agent --> builder-agent[Builder Agent]
  github-state-engine --> deployment-pipeline[Deployment Pipeline]
  deployment-pipeline --> sentry[Sentry]
  sentry --> incident-responder[Incident Responder]
  incident-responder --> github-state-engine[GitHub State Engine]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`GitHub State Engine`	Issues, PRs, and workflow state for the factory.	`Builder Agent`, `Reviewer Agent`, `Incident Responder`
`Builder Agent`	Implements features, writes tests, and updates Memo or the factory.	`GitHub State Engine`, `Test Runner`
`Reviewer Agent`	Reviews code and requests fixes before merge.	`GitHub State Engine`, `Builder Agent`
`Test Runner`	Runs unit and end-to-end tests and feeds failures back to agents.	`Builder Agent`, `Reviewer Agent`
`Deployment Pipeline`	Deploys accepted changes.	`GitHub State Engine`, `Sentry`
`Sentry`	Observability signal for incidents and bugs.	`Incident Responder`
`Incident Responder`	Creates or fixes issues based on production signals.	`Sentry`, `GitHub State Engine`

Data flow

GitHub issues and PRs hold factory state. Builder agents select tasks, implement changes, write or extend tests, and submit pull requests. Reviewer agents inspect changes and request fixes. Deployment and Sentry feedback create new issues or incidents that the factory can pick up and route back into the same loop.

“your engine of state is GitHub basically. Yeah. Or, you know, for some of these issues and things. But, then because that environment has access to GitHub, it can run the CLI and because it's authenticated, etc.”transcript, stated

“Sentry, the incident responder will pick it up, will assess it, triage it, and if it's a verified bug, it will open an issue that the bug fixer then um, will pick up. And the other option is that”transcript, stated

Why this architecture

Using GitHub as the state manager let the factory reuse the same primitives human teams already use for tasks, pull requests, and reviews. The experiment also treats tests and observability as first-class because autonomous code generation without review and incident feedback would not be operationally meaningful.

“GitHub in order to be effectively a state manager for the factory itself. >> Which was also a design decision. Um I think that's worth noting because if you look at, for example, a lot of the the harnessing that's going on,”transcript, stated

“didn't have tests. So, it started writing tests as well when it didn't have anything to do. And then on Monday, uh at the end of the day, um I added something to the spec, which was the database.”transcript, stated

Models used: AI agents

Scale

The experiment ran as a two-week software-factory series.
“summit. 2 weeks in software factory. Um yes, welcome back everyone. What How's it going Zach? How are you? >> Very good. Very good. Very pumped about this.”transcript, stated
Around 88% of pull requests were completed autonomously at the moment discussed.
“without a human in the loop. And we are at the moment I think around 88% of PRs completely done by agents and that means around 12% were either triggered by a human in that case me”transcript, stated
The factory worked on Memo, a Notion-like application.
“Notion would, you know, look like or what features Notion has and identify gaps to come to this product. Uh and then added some of the functionality here. As well as And also uh identified that, you know, a lot of the components”transcript, stated

Alternatives considered (roads not taken)

Build Memo directly by hand.

Rejected because: The product being tested was the factory, not only the demo app.

“Because while the factory built Memo, our product was actually the factory. And also, for for me the last 2 weeks, I think that was the the biggest biggest learning here is my job was, you know, obviously looking”transcript, stated

Custom state manager outside GitHub.

Rejected because: The team was already operating through GitHub issues and pull requests.

“use GitHub, you know, for all the issues that are in progress, for everything that's done, you can just query GitHub”transcript, stated

Code generation without review.

Rejected because: Review became one of the most scrutinized and important stages.

“Having the agent do its own code review, and you know, naturally it's it's a different agent doing a scrutinized code review”transcript, stated

Patterns

`tool-use-orchestration` (software-factory)

Agents coordinate GitHub, tests, review, deploys, and Sentry-driven operations.

“building and reviewing is almost the”transcript, implied

`observability-tracing` (operations)

Sentry provides incident feedback that the factory can route into fixes.

“Sentry, the incident responder will pick”transcript, stated

`eval-driven-development` (quality)

Tests and review loops drive whether agent-generated code proceeds or needs fixes.

“lead to more loops uh in in the review versus build phase and you know, the reviewer says okay, there's more to fix”transcript, stated

Tradeoffs

State management

Chose GitHub as source of truth over Separate factory database.

Rationale: GitHub already managed issues, PRs, CLI access, and workflow state.

“GitHub as the source”transcript, stated

Factory target

Chose Build the factory while it builds Memo over Only ship the Notion-like app.

Rationale: The experiment's value was proving the repeatable factory pattern.

“don't build Memo, you build a factory to build Memo.”transcript, stated

Gotchas (what bit them)

Autonomous feature work exposes missing test infrastructure.

Root cause: Agents need feedback loops, and the project initially lacked enough tests.

Resolution: Have the factory write unit and end-to-end tests when gaps appear.

“didn't have enough testing going on.”transcript, stated

The demo app can distract from the real product under test.

Root cause: Memo was the output, but the durable asset was the factory process.

Resolution: Focus fixes and learning on factory capability, not only Memo features.

“fixes were actually not on Memo, but they were on the factory”transcript, stated

Evaluation

The experiment evaluated autonomous PR completion, test growth, review loops, deployability, production incident response, and whether the factory could improve itself.

Metrics: Autonomous PR completion · Test coverage growth · Review loops · Deployment state · Sentry incidents · Factory self-improvement

“Pretty much autonomously”transcript, stated

Outcomes

The factory built Memo over a two-week public experiment.
Around 88% of PRs were completed autonomously.
The factory wrote tests and used GitHub/Sentry loops for maintenance.

The source cites a two-week series, around 88% autonomous PR completion, GitHub as state manager, and Sentry-driven incident response.

“factory has built itself.”transcript, stated