Harness Engineering: Structuring Context and Guardrails for AI Coding Agents in Production

openai · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: production coding-agent harnesses

OpenAI needed a way to make AI coding agents productive in real software work even though agents lack human cultural osmosis, durable memory, standup context, and implicit quality norms, while human attention and model context remain scarce.

Constraints:

• modern coding agents can produce PRs and handle parts of the software lifecycle, so implementation is no longer the only bottleneck
• human time, human/model attention, and model context windows remain foundational limits
• agents do not naturally accumulate team culture, code-review scars, or durable memory from meetings and standups
• quality requirements must be written down and surfaced when the agent needs them
• front-loading too many guardrails can thrash the model and reduce creativity
• long-running sessions can span many auto-compacted context windows
• review feedback should not be repeated manually if it can become a static guardrail
• teams must decide when to keep interventions late in the process and when repeated failures justify shifting them left

Architecture

The talk proposes a harness where humans define quality in text, agents.md points agents through planning and grounding steps, documentation and ADRs supply product and codebase context, tests/lints/tool outputs inject just-in-time remediation, static guardrails make repeated failures impossible, reviewer-agent personas judge PRs through specific quality lenses, and feedback from reviews, failed builds, interventions, and production exceptions is captured for later guardrail improvements.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  human_operator["Human operator"] --> agents_md["Agents.md"]
  human_operator --> feedback_capture_loop["Feedback capture loop"]
  agents_md --> coding_agent["Coding agent"]
  documentation_and_adrs["Documentation and ADRs"] --> coding_agent
  review_persona_guardrails["Review persona guardrails"] --> reviewer_agents["Reviewer agents"]
  agents_md --> review_persona_guardrails
  coding_agent --> tool_and_test_outputs["Tool and test outputs"]
  tool_and_test_outputs --> static_guardrails["Static guardrails"]
  static_guardrails --> coding_agent
  static_guardrails --> reviewer_agents
  coding_agent --> pr_review_thread["PR review thread"]
  reviewer_agents --> pr_review_thread
  pr_review_thread --> human_operator
  pr_review_thread --> feedback_capture_loop
  feedback_capture_loop --> sub_agent_distillation["Sub-agent distillation"]
  sub_agent_distillation --> documentation_and_adrs
  sub_agent_distillation --> static_guardrails
  codex_app_automation["Codex app automation"] --> coding_agent
  human_operator --> codex_app_automation

Components

Data flow

An agent starts from a ticket and agents.md, grounds itself in documentation, ADRs, design docs, and critical user journeys, then enters implementation while tools, tests, lints, and runbook-linked errors inject context at the point of failure. After the diff exists, static guardrails and LLM-as-judge reviewer agents assess reliability, performance, and team standards in the PR thread. Repeated review comments, failed builds, production exceptions, and interruptions are captured as evidence of missing context and distilled into new docs, guardrails, and tests.

Why this architecture

The harness is designed around the fact that agents need text and context, but attention and context windows are scarce. By making team standards explicit, surfacing them at the right moment, and progressively encoding repeated mistakes as static guardrails, the system aims to reduce synchronous human attention while preserving quality.

Models used: o3-era reasoning models · Codex CLI · GPT 5.2 · Opus 4.5 · reviewer agents · LLM-as-judge evaluators · sub-agents · Codex app

Scale

• The speaker says he personally maxes out at about three concurrent agent sessions on his laptop.
  “I know I max out probably at three concurrent sessions on my laptop.”transcript, stated
• Long GPT-series agent tasks can run for six, twelve, or thirty-six hours with auto-compaction.
  “I can let a task go for six, 12, 36 hours and still get good results”transcript, stated
• The talk frames the model shift around GPT 5.2 and Opus 4.5 in December.
  “in December, with the introduction of GPT 5.2, Opus 4.5, we really reach singularity levels of software engineering”transcript, stated
• A single prompt can now produce one PR or six PRs.
  “we can give a prompt to a coding agent and get a PR or six out of it”transcript, stated
• The guardrail example uses 100% code coverage and exhaustive table-driven tests.
  “we can absolutely vibe a set of guardrails into place with 100% code coverage and exhaustive table driven tests”transcript, stated
• The snapshot-test example requires 100% branch coverage for every React component.
  “every react component in our codebase has a snapshot test that gives 100% branch coverage”transcript, stated
• The Q&A uses tasks spanning fifteen context windows as the signal for shifting a guardrail left.
  “guardrails are commonly required over tasks that span 15 context windows”transcript, stated

Alternatives considered (roads not taken)

Rely on human cultural osmosis and repeated review feedback

Rejected because: Agents do not attend standups or retain battle scars, so teams must make quality rules explicit and retrievable.

Only write requirements in documentation

Rejected because: Documentation has no effect unless the right text enters the agent context at the time the agent needs it.

Shift every quality intervention as far left as possible

Rejected because: For agent work, the speaker starts with interventions as far right as possible to minimize synchronous human time, then shifts left only when needed.

Use one-off prompt fixes for bad PRs

Rejected because: Prompt fixes can work locally but do not reliably socialize the improvement to the team or future agents.

Jam many guardrails directly into agents.md

Rejected because: Too many prescriptive rules chop up the model's latent space and make creative codebase exploration harder.

Treat local guardrail tooling as production-quality software

Rejected because: For local-only harnesses, rough code is acceptable if it enforces the right invariants and reduces human attention.

Patterns

context-engineering (proposed harness discipline)

The talk defines harness engineering as making quality context legible and surfacing it just in time through agents.md, docs, tests, lints, tool outputs, review personas, and auto-compaction-aware guardrail placement.

tool-use-orchestration (coding-agent execution loop)

The agent uses tools, tests, lints, computer/browser capabilities, Codex app automation, and PR collaboration surfaces to execute and prove work.

eval-driven-development (guardrail and review loop)

Quality control is encoded as tests, lints, snapshot tests, branch coverage, static guardrails, LLM-as-judge review, and reviewer-agent feedback that the implementation agent must satisfy.

human-in-the-loop-gate (merge and intervention policy)

Humans retain merge judgment, decide which issues are P2 or higher, ask agents for proof such as logs/screenshots, and shift guardrails left when repeated problems appear.

safety-policy-enforcement (static quality enforcement)

The system turns reliability, type-safety, retries/timeouts, snapshot coverage, branch coverage, and other non-functional requirements into static checks that prevent recurrence.

multi-agent-orchestration (feedback distillation)

The future loop points sub-agents at accumulated human feedback, failed builds, and production exceptions to identify better prompts and missing guardrails.

Tradeoffs

Move quality gates as far right as practical.

Chose Late interventions first, then shift left only when repeated failures require it over Default DevOps shift-left behavior for every concern.

Rationale: The speaker wants to minimize synchronous human time and rely on right-time discovery until a pattern proves it needs stronger earlier enforcement.

Keep context timely rather than exhaustive.

Chose Just-in-time prompt injection through tools, tests, lints, and PR review over Large upfront context dumps.

Rationale: Just-in-time context reduces context-window pollution and lets the model reason creatively before receiving targeted constraints.

Make repeated mistakes impossible.

Chose Static guardrails, lints, exhaustive tests, and reviewer agents over Repeated manual review comments.

Rationale: Every repeated feedback item is evidence that context was missing and should be encoded durably.

Use review personas instead of one monolithic rule file.

Chose Curated persona-oriented guardrail files referenced by agents.md over Jamming all guardrails directly into the central agent instruction file.

Rationale: Persona files let agents discover relevant guardrails without over-constraining every task.

Bias review toward merge with P2+ blockers.

Chose Benefit-of-the-doubt PR acceptance with reviewer agents surfacing P2-and-above issues over Shoulder-surfing every agent-generated change.

Rationale: The process treats agents like teammates and focuses human attention on merge-blocking issues and proof artifacts.

Gotchas (what bit them)

Agents lack human team osmosis.

Root cause: They do not attend standups or accumulate durable team memory and battle scars like human engineers.

Resolution: Write down non-functional requirements and make them legible, retrievable, and timely inside the harness.

Documentation alone does not steer an agent.

Root cause: The text must be pulled into context at the moment the agent is making the relevant decision.

Resolution: Use just-in-time context injection through tools, tests, lints, runbooks, and review agents.

Point-in-time fixes do not compound.

Root cause: Changing a prompt or giving a one-off review comment may solve one PR but does not reliably socialize the improvement to the team.

Resolution: Convert repeated feedback into docs, reviewer agents, lints, guardrails, and tests applied to every future PR.

Too many upfront rules can damage reasoning.

Root cause: Jamming a large rule set into context chops up latent space and makes creative codebase exploration harder.

Resolution: Keep agents.md as a map to relevant guardrails and rely on persona files and auto-discovery when possible.

Auto-compaction can erase guardrail context on long tasks.

Root cause: Tasks spanning many context windows may lose guardrail files that were originally paged in.

Resolution: Use repeated failures across long tasks as a signal to shift those guardrails left into stronger checks.

Agent claims of testing need proof just like human claims.

Root cause: The reviewer cannot shoulder-surf every agent or teammate, so acceptance depends on artifacts that demonstrate behavior.

Resolution: Ask for logs, screenshots, browser/computer-use evidence, or reproduction videos attached to the PR.

Evaluation

The harness is evaluated by observing remaining synchronous human attention, repeated review comments, failed builds, production exceptions, guardrail misses, proof artifacts attached to PRs, and whether agents can run long tasks through multiple context windows without losing necessary quality constraints.

Metrics: three concurrent sessions as the speaker's personal synchronous-attention limit · agent tasks can run six, twelve, or thirty-six hours with auto-compaction · PRs can be one or six from a single prompt · 100% code coverage for generated guardrails in the retry/timeout example · 100% branch coverage for every React component snapshot-test requirement · P2-and-above issues are the review threshold for merge acceptance · fifteen context windows is cited as a signal that a guardrail may need to shift left

Outcomes

• The speaker moved from an early failed Slack-alert triage attempt to a harness approach based on tools, context, and guardrails.
• The process reframes human software engineers as operators who unblock execution teams of coding agents.
• Quality control becomes a feedback system: observe missing context, encode it in docs/tests/guardrails/reviewer agents, and avoid giving the same feedback twice.
• Static guardrails can make reliability and type-safety errors impossible rather than merely discouraged.
• The speaker reports applying these techniques to open-source Rust crate maintenance through Codex app automations, with review agents still planned.

The talk cites June-era o3/Codex CLI experiments, GPT 5.2 and Opus 4.5 in December, three concurrent local sessions, six-to-thirty-six-hour tasks, one-to-six PRs per prompt, 100% code/branch-coverage guardrails, and tasks spanning fifteen context windows.