Extreme Harness Engineering: Building Production Systems with Zero Human-Written Code

openai · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Agentic production-system development

OpenAI Frontier explored whether an internal production system could be built and operated with zero human-written code by making Codex agents responsible for implementation, review, merge flow, observability, and repository evolution.

Constraints:

• The experiment started from a hard constraint that humans would not write the application code.
• The agent needed enough context, tools, and building blocks to do end-to-end engineering work.
• Parallel coding agents made human attention and context switching the limiting factor.
• Autonomous PR work required fast builds, tests, CI handling, review feedback, and merge-conflict resolution.
• Agents needed software, docs, skills, and dashboards designed for agent legibility.
• Enterprise deployment required observable, safe, controlled, and identifiable agents.

Architecture

The system combines Codex, repository skills, text-first tooling, observability, fast build/test loops, agent review, worktrees, GitHub/CI automation, and Symphony. Humans define goals and encode constraints, agents author and review code, Symphony coordinates many concurrent tasks and rework loops, and release promotion remains gated by human smoke tests. The broader Frontier framing extends the same principles to observable enterprise agents.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  human-product-driver["Human product driver"] --> codex-harness["Codex harness"]
  codex-harness --> coding-agent-workers["Coding-agent workers"]
  agent-readable-repository-context["Agent-readable repository context"] --> codex-harness
  text-first-tooling["Text-first tooling"] --> codex-harness
  local-observability-and-dashboards["Local observability and dashboards"] --> text-first-tooling
  coding-agent-workers --> fast-build-and-ci-loop["Fast build and CI loop"]
  coding-agent-workers --> pr-automation-skill["PR automation skill"]
  review-agent-workers["Review-agent workers"] --> pr-automation-skill
  pr-automation-skill --> worktree-execution-pool["Worktree execution pool"]
  worktree-execution-pool --> symphony-coordination-service["Symphony coordination service"]
  symphony-coordination-service --> release-smoke-test-gate["Release smoke-test gate"]
  release-smoke-test-gate --> human-product-driver
  review-agent-workers --> knowledge-distillation-loop["Knowledge distillation loop"]
  knowledge-distillation-loop --> agent-readable-repository-context

Components

Data flow

Tasks enter as tickets, specs, or prompts; Codex agents read repository skills and context, run scripts and local observability, change code, open PRs, receive agent/human review, fix CI and merge conflicts, enter merge queues, or get restarted by Symphony's rework state. Feedback from PR comments, failures, and trajectories is distilled back into skills, docs, tests, and repository context.

Why this architecture

The team optimized the whole production system for agent autonomy: if a failure showed missing capability, context, or structure, they encoded it into tools, docs, tests, skills, observability, or orchestration rather than asking humans to supervise more. Symphony exists because PR context switching became taxing once agent throughput reached multiple PRs per engineer per day.

Models used: Codex CLI · Codex Mini · Codex · GPT-5.2 · GPT-5.3 · GPT-5.4 · Symphony · Elixir · Beam · Review agents · GitHub CLI · Playwright shim CLI

Scale

• The experiment built an internal tool over five months with zero human-written code and about one million lines.
  “five months of working on an internal tool, zero lines of code over a mi, a million lines of code in the total code base.”article, stated
• The team was producing around 1,500 PRs when human review became a bottleneck.
  “You kicked off as a team of three people. You’re putting out a million line of code, like 1500 prs, basically.”article, stated
• Throughput reached five to ten PRs per engineer per day.
  “We were up in the five to 10 PRS per day per engineer.”article, stated
• The system used repository decomposition more like a 10,000-engineer architecture than a seven-person team.
  “that this is fundamentally why we have such a rigid, like 10,000 [00:37:00] engineer level architecture in the app”article, stated
• The codebase used roughly 500 NPM packages.
  “The structure of the repository is like 500 NPM packages.”article, stated
• The team operated at roughly a billion tokens of intelligence per day.
  “you deploy a billion tokens of intelligence a day”article, stated

Alternatives considered (roads not taken)

Let humans remain the primary code authors.

Rejected because: The experiment's purpose was to discover whether agents could do all the things the human engineer did.

Monitor every agent run synchronously.

Rejected because: The team wanted to build systems that went the right way without humans babysitting agents.

Keep failed generated code and patch around it.

Rejected because: Symphony treats code as cheap and can restart failed PR work after diagnosing the missing context.

Expose broad MCP tools directly.

Rejected because: The source reports that broad tool schemas cost too many context tokens and could interfere with auto-compaction.

Build an off-policy scaffold that constrains Codex output.

Rejected because: The preferred guardrails are native to code and tests, so they do not fight model progress.

Patterns

multi-agent-orchestration (PR and task coordination)

Symphony coordinates many coding-agent tasks, worktrees, reviews, rework loops, and PR lifecycle steps.

durable-execution (Elixir process supervision)

The orchestration service uses Elixir/Beam process supervision to represent each active task as a daemon that can be driven to completion.

observability-tracing (agent-debuggable production systems)

Agents can inspect traces, metrics, logs, dashboard definitions, and alert context, then fix missing instrumentation.

tool-use-orchestration (text-first agent operations)

Agents use token-efficient CLIs, scripts, and shims rather than human-oriented GUIs or giant logs.

agent-memory (team knowledge distillation)

Agent trajectories and session logs are distilled into a team-level knowledge base and reflected back into code and repository guidance.

human-in-the-loop-gate (release governance)

Humans approve smoke tests and release promotion even when code authorship and review are heavily delegated.

Tradeoffs

How to get production-system leverage from agents.

Chose Encode taste into context, tools, observability, tests, and orchestration over Manual code authorship or one-off prompting.

Rationale: Durable context lets many agents reuse team knowledge and improves future runs rather than only fixing one task.

How to handle concurrency across many agents and humans.

Chose Deep decomposition, sharding, and interface boundaries over Small-team repository structure.

Rationale: When each human controls many agents, team size behaves like a much larger engineering organization.

How much code review detail the human should own.

Chose Representative sampling and system-level inference over Deep review of every PR.

Rationale: The human acts more like a group tech lead for a large organization, using samples to identify systemic problems.

How to distribute Symphony-like software.

Chose Ghost-library/spec-driven distribution over Shipping all internal implementation source.

Rationale: A high-fidelity spec can let a coding agent reassemble a complex system locally.

Gotchas (what bit them)

The first month and a half was slower than human work.

Root cause: Early Codex versions and the initial harness lacked enough primitives for the agent to assemble product features.

Resolution: Invest in tools, building blocks, and the assembly station for agent work instead of reverting to manual coding.

Build time became an agent-productivity constraint.

Root cause: Background shells changed the model's patience for blocking scripts and long-horizon work.

Resolution: Switch build systems until the inner loop completed in under a minute.

Review feedback can create non-convergent agent loops.

Root cause: The authoring agent followed reviewer-agent comments even when they over-expanded scope.

Resolution: Instruct reviewer agents to bias toward merging and authoring agents to defer or push back where appropriate.

Human-oriented debugging tools can keep engineers unnecessarily in the loop.

Root cause: A human built a trace visualization app even though Codex could answer directly from the trace tarball.

Resolution: Prefer agent-readable artifacts and ask Codex to inspect raw debug material when possible.

Current models still struggle with zero-to-one products and gnarly refactors.

Root cause: Translating whitespace product intent and deep interface shape into context still requires human steering.

Resolution: Use harnesses and scaffolds to extract missing context, and expect humans to spend time on the hardest white-space/refactoring problems.

Evaluation

The system is evaluated through throughput, build speed, CI, tests, review-agent output, merge success, post-merge sampling, smoke tests, observability gaps, and whether failures turn into better repository context.

Metrics: Five to ten PRs per engineer per day · Inner loop under one minute · CI green status · Merge conflict handling · Review-agent feedback priority · Human-approved smoke tests · Agent trajectory distillation

Outcomes

• The team built an internal Electron/native product with zero human-written code.
• Humans moved out of routine authorship and review, with agent review, PR lifecycle automation, and post-merge sampling taking over much of the loop.
• Symphony emerged as a coordination system for large numbers of coding agents.
• The approach reframed production systems around agent-readable context, observability, and orchestration.

The source reports a five-month, million-line, roughly 1,500-PR experiment; five to ten PRs per engineer per day; a roughly 500-package repository; and billion-token-per-day operation.