Extreme Harness Engineering: Building Production Software with Zero Human-Written Code

openai · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI-native software engineering

OpenAI Frontier Product Exploration wanted to test whether production software could be built with zero human-written code, which shifted the bottleneck from model tokens to harness design, context, observability, task decomposition, agent coordination, and scarce human attention.

Constraints:

• The experiment deliberately forbade the human engineer from writing application code.
• Early Codex models were much less capable and initially struggled to assemble product features.
• Builds, tests, review, merge conflicts, and release steps had to be legible and executable by agents.
• Humans became the scarce resource as agents could run in parallel with cheap tokens.
• The repository had to be structured for agent legibility and decomposition, not just human habits.
• Production release still required human-approved smoke testing before distribution.

Architecture

The team developed a harness-engineering workflow around Codex agents. Instead of writing code, humans decomposed failures into smaller building blocks, encoded taste into skills, docs, tests, markdown trackers, guardrails, and observability, kept build loops under a minute, delegated PR lifecycle work to Codex skills and review agents, used worktrees and CI/merge automation, and built Symphony, an Elixir orchestration layer that supervises and reworks large numbers of coding agents.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  frontier-product-exploration-team["Frontier Product Exploration team"] --> codex-harness["Codex harness"]
  codex-harness --> coding-agents["Coding agents"]
  repository-skills-and-docs["Repository skills and docs"] --> codex-harness
  local-observability-stack["Local observability stack"] --> codex-harness
  coding-agents --> build-and-test-system["Build and test system"]
  coding-agents --> pr-lifecycle-skill["PR lifecycle skill"]
  review-agents["Review agents"] --> pr-lifecycle-skill
  pr-lifecycle-skill --> worktree-and-merge-queue["Worktree and merge queue"]
  worktree-and-merge-queue --> symphony-orchestrator["Symphony orchestrator"]
  symphony-orchestrator --> human-release-gate["Human release gate"]
  review-agents --> team-knowledge-distillation["Team knowledge distillation"]
  team-knowledge-distillation --> repository-skills-and-docs
  human-release-gate --> frontier-product-exploration-team

Components

Data flow

A human defines or selects work, Codex agents use repository skills, docs, scripts, and local observability to author changes, a PR lifecycle skill pushes work through review, CI, merge conflict resolution, and merge queue handling, review agents and tests provide feedback, Symphony can rework failed PRs from scratch, and humans inspect compressed evidence plus smoke-test release branches before distribution.

Why this architecture

The architecture optimizes for agent legibility because agents can be parallelized cheaply while human attention cannot. The team therefore encoded taste, guardrails, and observability into text, tests, scripts, dashboards, and skills so agents could see, fix, review, and merge work with minimal human supervision.

Models used: Codex CLI · Codex Mini · Codex · GPT-5 model generations · Symphony · Elixir · Beam · Review agents · Skills · Grafana dashboards · Playwright shim CLI · GitHub CLI

Scale

• The experiment ran for five months with zero lines of human-written code and roughly a million-line codebase.
  “five months of working on an internal tool, zero lines of code over a mi, a million lines of code in the total code base.”article, stated
• The repository had about 1,500 PRs when humans became the review bottleneck.
  “You kicked off as a team of three people. You’re putting out a million line of code, like 1500 prs, basically.”article, stated
• Early work was about 10 times slower than the human engineer before harness investment paid off.
  “Honestly, the first month and a half was. 10 times slower than I would be.”article, stated
• The build inner loop was retooled to complete in under a minute.
  “we had to retool the entire build system to complete in under a minute”article, stated
• After GPT-5.2, the team reached about five to ten PRs per engineer per day.
  “We were up in the five to 10 PRS per day per engineer.”article, stated
• The repository was decomposed into roughly 500 NPM packages.
  “The structure of the repository is like 500 NPM packages.”article, stated
• The team used about six repository skills for shared agent behavior.
  “So like in our code base, we have, I think, six skills. That’s it.”article, stated

Alternatives considered (roads not taken)

Have humans write code directly.

Rejected because: The experiment intentionally constrained the human to make the agent do the whole job and expose missing harness capabilities.

Prompt the agent harder when it fails.

Rejected because: The team instead built missing building blocks, tools, context, and structure so future agents could complete the task.

Keep humans synchronously reviewing all code.

Rejected because: Human attention became the bottleneck, so reviews moved toward agent review, post-merge sampling, and compressed evidence.

Use a predefined scaffold that boxes in the reasoning model.

Rejected because: The team found reasoning models performed better when the harness itself gave options and enough context to choose how to proceed.

Depend on visual GUI-oriented tools for agent debugging.

Rejected because: Agents work better with text and token-efficient CLIs, so the team converted visual and noisy tools into text-friendly shims.

Patterns

multi-agent-orchestration (coding-agent coordination)

Symphony supervises many coding-agent tasks, PRs, worktrees, and rework states so humans do not manually drive every terminal session.

observability-tracing (agent-visible operations)

The team gave Codex traces, metrics, logs, dashboards, pages, and local observability so agents could diagnose and fix issues.

durable-execution (agent task supervision)

Symphony uses Elixir process supervision and task daemons to drive agent work to completion and rework failed PRs.

human-in-the-loop-gate (release control)

Humans stay out of routine code authorship but still approve release branches and smoke tests before distribution.

tool-use-orchestration (agent-operable developer tools)

The workflow exposes boot scripts, CLIs, GitHub commands, dashboards, and small shims as tools agents can invoke.

eval-driven-development (self-improving engineering loop)

Failed builds, PR comments, review feedback, and agent session logs become signals that missing context or requirements should be encoded back into the repo.

Tradeoffs

How to spend human effort under the zero-human-code constraint.

Chose Build harness, tools, context, and observability over Writing or reviewing every code change directly.

Rationale: The leverage came from making the agent able to do the whole job, while humans focused on the next bottleneck.

How to structure the repository.

Chose Agent-legible, highly decomposed architecture over Normal human-scale team structure.

Rationale: If each person effectively commands many agents, deep decomposition and stable interfaces reduce collisions and dispersion.

How to handle failed Symphony PRs.

Chose Trash and rework from scratch after diagnosis over Patch the same bad worktree indefinitely.

Rationale: Code is cheap enough that the valuable step is identifying what context or requirement was missing before retrying.

How to encode guardrails.

Chose Native tests, docs, scripts, and code-level constraints over A separate off-policy scaffold around Codex.

Rationale: Native code/test guardrails align with what Codex already produces and improve reliability without fighting future model improvements.

Gotchas (what bit them)

Early models could not assemble product features without better primitives.

Root cause: The initial harness lacked the building blocks and structure the model needed.

Resolution: Double-click failures into smaller building blocks that can be reassembled into broader objectives.

Model and harness changes can invalidate assumptions about long-running scripts.

Root cause: Background shells made the model less willing to block on long tasks.

Resolution: Retool the build system so the inner loop completed in under a minute.

Review agents and authoring agents can thrash if feedback is treated as mandatory.

Root cause: The authoring Codex was initially willing to be bullied by PR reviewer comments.

Resolution: Give reviewer agents priority guidance and authoring agents permission to acknowledge, defer, or push back.

MCP-style broad tool exposure can waste context and hurt compaction.

Root cause: The harness injected too many tool tokens and the agent only needed a few Playwright calls.

Resolution: Build a tiny local daemon and shim CLI that exposes only the needed operations.

Multi-human, multi-agent work can become chaotic without shared state.

Root cause: Humans are not continuously in the loop and can lose track of what the code currently looks like.

Resolution: Use rigid decomposition, daily synchronous knowledge sharing, and team-level knowledge distillation.

Evaluation

The team evaluated the harness through build times, PR throughput, CI, review-agent feedback, post-merge review, observability dashboards, smoke tests, and signals from failed builds or PR comments that indicated missing context.

Metrics: Build loop under one minute · PRs per engineer per day · CI pass/fail · Review-agent comments · Post-merge human review samples · Smoke-test approval before distribution · Agent trajectory and session-log distillation

Outcomes

• The team built an internal beta product with zero human-written code.
• The codebase reached about one million lines and roughly 1,500 PRs.
• The workflow moved from slower-than-human early Codex work to five to ten PRs per engineer per day.
• Symphony emerged as an Elixir coordination layer for large numbers of coding agents.
• The team shifted human work toward system design, observability, context, release gates, and next-bottleneck discovery.

The source reports a five-month experiment, a million-line codebase, around 1,500 PRs, a repository structure near 500 NPM packages, six shared skills, and billion-token-per-day operating scale.