Harness Engineering: Building Software Where Humans Steer and Agents Execute

openai · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI-assisted software engineering and coding-agent operations

OpenAI's harness engineering approach addresses how to make coding agents do production software work when implementation capacity is abundant, but human time, model attention, context windows, review capacity, and non-functional requirements remain scarce.

Constraints:

• human engineers were required to work through agents rather than directly editing code
• implementation is no longer the scarce resource, but human time, human and model attention, and context window are scarce
• agents must satisfy hundreds of non-functional quality decisions that are usually implicit in human engineering practice
• context pages out over time, so important instructions cannot all be front-loaded
• high agent-driven PR volume creates merge conflicts and makes human review the bottleneck
• skills, local developer tools, and repository infrastructure change frequently
• large repositories need progressive disclosure, package boundaries, uniform patterns, and context-efficient code layout
• the team must preserve quality for security, reliability, QA, smoke tests, PII controls, runbooks, and operational workflows beyond code generation

Architecture

The system turns the repository, CI, local tools, and collaboration process into a harness around coding agents. Tickets enter through Codex, skills teach the agent how to launch apps and observability tools, documentation and ADRs define what good looks like, lints/tests/review agents inject just-in-time instructions, repository architecture makes context local and uniform, and humans move from typing code to defining work, guardrails, acceptance criteria, and higher-leverage process improvements.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  human_engineer["Human engineer"] --> ticket_backlog["Ticket backlog"]
  ticket_backlog --> codex_entry_point["Codex entry point"]
  codex_entry_point --> skills_layer["Skills layer"]
  skills_layer --> local_app_harness["Local app harness"]
  skills_layer --> observability_stack["Observability stack"]
  local_app_harness --> chrome_devtools_and_daemons["Chrome DevTools and daemons"]
  codex_entry_point --> repository_architecture["Repository architecture"]
  documentation_and_standards["Documentation and standards"] --> review_agents_and_ci["Review agents and CI"]
  repository_architecture --> review_agents_and_ci
  observability_stack --> review_agents_and_ci
  review_agents_and_ci --> github_pr_workspace["GitHub PR workspace"]
  codex_entry_point --> github_pr_workspace
  github_pr_workspace --> human_engineer
  human_engineer --> garbage_collection_loop["Garbage collection loop"]
  garbage_collection_loop --> documentation_and_standards
  garbage_collection_loop --> review_agents_and_ci

Components

Data flow

A human defines or schedules a ticket, gives it to Codex with a small set of high-leverage skills, and lets the agent operate inside a repository designed for agent use. As the agent edits code, CI, custom lints, source-structure tests, review agents, QA-plan requirements, and PR comments inject the relevant standards at the point of need. Human review feedback is converted into documentation and automated checks during garbage-collection cycles, which reduces repeated slop and lets later agent runs self-correct.

Why this architecture

The harness exists because models can produce abundant code, but they need timely, local, explicit instructions about quality, context, tools, and acceptance criteria. Instead of relying on human reviewers to remember every non-functional requirement, the team moves those requirements into docs, lints, tests, review agents, repo structure, and skills so agents receive corrective context while doing the work.

Models used: GPT-5.2 · GPT-5.4 · Codex · coding agents · review agents · agent SDKs embedded in tests

Scale

• The speaker reports nine months of building software exclusively with agents.
  “for the last 9 months, I have had the privilege of building software exclusively with agents.”transcript, stated
• Each engineer is framed as having access to between five and 5,000 engineers worth of agent capacity.
  “each engineer today in this room has access to five 50 or 5,000 engineers worth of capacity 24/7 every day of the year.”transcript, stated
• The team uses file-size tests that keep files under 350 lines for context efficiency.
  “we can write a test that limits the fact that files are no longer than 350 lines.”transcript, stated
• The repository architecture grew to about 750 PNPM packages.
  “750 packages in the PNPM workspace, isolated by business logic domain or layer of the stack”transcript, stated
• A small team was producing three to five PRs per engineer per day.
  “with each engineer producing three to five PRs per day, even on a team of three, merge conflicts were super miserable”transcript, stated
• The speaker describes spending over a billion output tokens per day, costing over a thousand dollars.
  “He's a token billionaire spending over a billion output tokens a day. That's like over a thousand dollars.”transcript, stated
• Token usage is roughly split one-third each across planning/documentation, implementation, and CI.
  “it's probably a third a third a third between like planning, ticket curation, documentation, implementation, and stuff that runs in CI.”transcript, stated

Alternatives considered (roads not taken)

Let humans continue typing implementation code directly

Rejected because: The team intentionally banned direct editor use to force the workflow through agents and move human effort toward systems thinking and delegation.

Build a shell environment around Codex

Rejected because: The team made Codex the outside-in entry point and taught Codex how to invoke the app, observability, and tooling itself.

Create thousands of narrow skills

Rejected because: Tooling changes too quickly for humans to maintain broad skill sprawl, so the team centralizes leverage around fewer better skills.

Front-load all instructions into the agent context

Rejected because: Front-loading overwhelms the agent and wastes scarce context, while just-in-time lints and tests provide requirements when they matter.

Use human code review as the main quality gate

Rejected because: At three to five PRs per engineer per day, humans became the blocker, so review personas, docs, tests, and lints moved review feedback into the harness.

Approve agent-generated plans without deep review

Rejected because: Unreviewed plans can encode bad instructions, so the speaker recommends plan-only PRs with line-by-line human review if plans are used.

Patterns

context-engineering (proposed harness discipline)

The case treats context as an engineered scarce resource: instructions are deferred, refreshed, injected through lints/tests/review agents, and encoded into repo structure so agents see the right text at the right time.

tool-use-orchestration (agent tool harness)

Codex is given skills and local tools for launching apps, observability, Chrome DevTools, daemons, tests, and PR workflows rather than being treated as only a code generator.

eval-driven-development (automated quality gates)

The team converts recurring review feedback into lints, tests, QA-plan expectations, source-structure checks, and reviewer agents that run during CI and PR workflows.

observability-tracing (agent-operable development environment)

The skills and local harness expose logging, telemetry, and browser/devtool inspection so agents can validate behavior while they work.

human-in-the-loop-gate (review and plan approval)

Humans move out of routine implementation but still review high-leverage artifacts such as plans, process definitions, priorities, and quality standards.

safety-policy-enforcement (non-functional requirement enforcement)

Security, reliability, package privacy, dependency edges, PII controls, retry/timeout rules, and other non-functional requirements are converted into durable automated checks.

Tradeoffs

Force software work through agents.

Chose Human steering and agent execution over Human engineers directly editing implementation code.

Rationale: The team wanted to operationalize agent capacity and move humans toward systems thinking, delegation, and defining quality.

Use just-in-time context delivery.

Chose Deferred instructions through lints, tests, and review agents over Loading all standards at the start of the task.

Rationale: Just-in-time delivery preserves context and gives agents constraints after they have enough task understanding to apply them.

Concentrate skills.

Chose Five to ten maintained skills over Thousands of narrow skills.

Rationale: Fewer, better skills hide changing infrastructure behind stable human-invoked interfaces while letting Codex adapt to details.

Architect the repository for agent locality.

Chose Many packages, package privacy, shared utilities, and uniform implementation patterns over A simpler broad repository where agents must infer boundaries repeatedly.

Rationale: Package boundaries and uniform code make the relevant directory subtree and expected token patterns more predictable to agents.

Bias PR collaboration toward throughput.

Chose Agents and humans can acknowledge, defer, or reject feedback over Every review comment must be addressed before progress.

Rationale: The system tries to avoid agents being boxed in by reviewers and instead biases toward accepted, functional code.

Spend tokens across the whole delivery system.

Chose Planning/documentation, implementation, and CI all consume substantial token budget over Using tokens mostly for code writing.

Rationale: The speaker frames getting code accepted and advancing the product as the expensive part, not merely producing implementation.

Gotchas (what bit them)

Telling agents not to produce slop is not enough.

Root cause: Quality depends on specifying hundreds of implicit non-functional engineering decisions that models have seen many possible versions of during training.

Resolution: Observe repeated slop, document the desired standard, and put guardrails, lints, tests, and review agents in place.

Context pages out during long agent work.

Root cause: Auto-compaction and long-running tasks mean agents may lose earlier instructions unless the harness refreshes requirements.

Resolution: Use review agents, lints, tests, and just-in-time feedback to continually resurface the right context.

Humans are inconsistent at remembering non-functional requirements.

Root cause: Reviewers forget or inconsistently enforce requirements such as retries, timeouts, secure interfaces, and source-code structure.

Resolution: Encode those requirements in bespoke lints, tests, and CI checks that run on every change.

Too many skills create maintenance drag.

Root cause: Local developer tools and repository infrastructure change too frequently for humans to track many narrow skills.

Resolution: Centralize leverage around five to ten skills and improve those instead of going wide.

High agent throughput can make merge conflicts and human review the bottleneck.

Root cause: Several engineers producing multiple large PRs per day kept PRs open long enough for conflicts to accumulate.

Resolution: Restructure the code, shorten PR open time, automate review personas, and remove repeated human review bottlenecks.

Plans can become harmful instructions if approved casually.

Root cause: A plan that nobody reads can encode bad rollout instructions that the agent will faithfully execute.

Resolution: If plans are used, treat them as reviewable PR artifacts and block execution until humans approve the plan line by line.

New lifecycle phases start at zero even after code generation works.

Root cause: QA, smoke testing, PII review, user-ops runbooks, and production issue triage require their own docs and tools.

Resolution: Write acceptance criteria and build agent-usable tools for each new software lifecycle capability as it emerges.

Evaluation

The team evaluates the harness by tracking where human attention is still required, whether PRs merge without slop or conflict, whether non-functional requirements are automatically enforced, whether agents can continue until tests are green, and whether more of the software lifecycle can be delegated through documented acceptance criteria.

Metrics: nine months of agent-only software building · three to five PRs per engineer per day · 750 PNPM packages · file-size tests enforcing no more than 350 lines · five to ten core skills · over one billion output tokens per day · roughly one-third tokens each across planning/documentation, implementation, and CI · goal state of 50 agents running 24/7 without interaction

Outcomes

• The team operated for nine months with software built exclusively through agents.
• Human effort shifted toward systems thinking, delegation, standards, harness design, and defining what good looks like.
• Review-agent and lint feedback reduced repeated slop and moved code review guidance from synchronous human comments into reusable repository context.
• Repository architecture, package boundaries, and uniform implementation patterns made agent work more local and predictable.
• High PR throughput exposed merge conflicts and human review as bottlenecks, motivating automated review personas and shorter PR open times.
• The long-term target is a token budget plus ranked work and metrics that lets machines continually advance a product without explicit human hands on the wheel.

The source reports a nine-month agent-only workflow, over a billion output tokens per day, three to five PRs per engineer per day, 750 PNPM packages, five to ten core skills, 350-line file limits, and an aspirational goal of 50 agents running 24/7.