Harness Engineering for Agentic Coding Systems

langchain · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: agentic coding harness optimization

LangChain needed to improve the reliability and benchmark performance of deepagents-cli without changing the underlying coding model, by engineering the surrounding prompts, tools, middleware, tracing, and execution flow that determine how the agent works.

Constraints:

• the underlying model, gpt-5.2-codex, was kept fixed
• Terminal Bench 2.0 imposes strict timeouts and automated scoring across 89 tasks
• models are black boxes, so failures must be understood through visible inputs, outputs, and traces
• agents tend to stop after superficial self-review instead of entering build-verify loops
• context discovery and search are error-prone in unseen environments
• agents can enter doom loops by repeatedly editing the same broken approach
• reasoning models can spend over twice as many tokens or time if reasoning budget is overused
• automated harness changes can overfit to benchmark tasks and regress generalization without human review

Architecture

LangChain optimized deepagents-cli by instrumenting every agent action in LangSmith, using Harbor and Daytona to run Terminal Bench tasks, analyzing traces through an agent skill, and then changing only the harness. The improved harness focused on system prompts that require build-verify-fix behavior, LocalContextMiddleware that injects environment and tooling context, PreCompletionChecklistMiddleware that forces verification before exit, LoopDetectionMiddleware that interrupts repeated broken edits, and a reasoning-budget strategy that spends more compute on planning and verification than implementation.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  terminal_bench_20["Terminal Bench 2.0"] --> harbor_orchestrator["Harbor orchestrator"]
  harbor_orchestrator --> daytona_sandboxes["Daytona sandboxes"]
  harbor_orchestrator --> deepagents_cli_agent_loop["deepagents-cli agent loop"]
  daytona_sandboxes --> deepagents_cli_agent_loop
  system_prompt["System prompt"] --> deepagents_cli_agent_loop
  localcontextmiddleware["LocalContextMiddleware"] --> deepagents_cli_agent_loop
  precompletionchecklistmiddleware["PreCompletionChecklistMiddleware"] --> deepagents_cli_agent_loop
  loopdetectionmiddleware["LoopDetectionMiddleware"] --> deepagents_cli_agent_loop
  reasoning_budget_policy["Reasoning budget policy"] --> deepagents_cli_agent_loop
  deepagents_cli_agent_loop --> langsmith_tracing["LangSmith tracing"]
  harbor_orchestrator --> langsmith_tracing
  langsmith_tracing --> trace_analyzer_skill["Trace analyzer skill"]
  trace_analyzer_skill --> human_verification["Human verification"]
  human_verification --> system_prompt
  human_verification --> localcontextmiddleware
  human_verification --> precompletionchecklistmiddleware
  human_verification --> loopdetectionmiddleware
  human_verification --> reasoning_budget_policy

Components

Data flow

Terminal Bench tasks are orchestrated by Harbor in Daytona sandboxes and executed by deepagents-cli. Every action, metric, and trace is sent to LangSmith. A trace analyzer skill fetches experiment traces, spawns parallel error-analysis agents, synthesizes findings, and proposes targeted harness changes. Humans may verify the suggested changes to avoid overfitting, then the next harness version is rerun against the benchmark and the loop repeats.

Why this architecture

LangChain chose harness optimization because a model's behavior in autonomous coding depends heavily on context, tools, prompts, middleware, verification loops, and reasoning-budget choices. Tracing makes failures observable at scale, while middleware and context injection compensate for current model tendencies such as weak self-verification, poor time estimation, environment-discovery errors, and doom loops.

Models used: gpt-5.2-codex · Claude Opus 4.6 · reasoning models · parallel error analysis agents · main synthesis agent

Scale

• The harness-only changes improved deepagents-cli by 13.7 points, from 52.8 to 66.5.
  “from 52.8 to 66.5 on Terminal Bench 2.0.”article, stated
• Terminal Bench 2.0 includes 89 tasks across domains such as machine learning, debugging, and biology.
  “It has 89 tasks across domains like machine learning, debugging, and biology.”article, stated
• The default prompt and standard tools plus middleware scored 52.8% with GPT-5.2-Codex.
  “We start with a default prompt and standard tools+middleware. This scores 52.8% with GPT-5.2-Codex.”article, stated
• Doom loops appeared as 10 or more small variations on the same broken approach in some traces.
  “that make small variations to the same broken approach (10+ times in some traces).”article, stated
• Maximum reasoning everywhere scored 53.9%, while high reasoning scored 63.6%.
  “Running only at xhigh scored poorly at 53.9% due to agent timeouts compared to 63.6% at high.”article, stated
• More reasoning can burn over twice as many tokens and time under Terminal Bench timeouts.
  “More reasoning helps agents evaluate each step, but can burn over 2x more tokens/time.”article, stated
• An earlier harness version scored 59.6% with Claude Opus 4.6.
  “A test run with Claude Opus 4.6 scored 59.6% with an earlier harness version”article, stated

Alternatives considered (roads not taken)

Improve performance by changing the underlying model

Rejected because: The experiment intentionally kept gpt-5.2-codex fixed and improved performance only by changing the harness.

Optimize every possible harness knob at once

Rejected because: LangChain deliberately compressed the design space to system prompts, tools, and middleware for this iteration.

Let the agent decide when its work is done

Rejected because: Agents naturally stopped after superficial self-review, so the harness added explicit build-verify-fix instructions and pre-completion verification middleware.

Trust the agent to discover its environment by search

Rejected because: Context discovery and search are error-prone, so the harness injects directory and tooling context up front.

Run every task at maximum reasoning

Rejected because: Maximum reasoning caused timeouts and lower score, so the team used a planning/implementation/verification reasoning sandwich.

Automatically apply trace-analyzer suggestions without human review

Rejected because: Humans are useful for preventing task overfitting and regressions in other tasks.

Patterns

context-engineering (proposed harness discipline)

LangChain explicitly treats context assembly and delivery as a harness responsibility, using LocalContextMiddleware, directory/tool discovery, time-budget warnings, testing criteria, and problem-solving strategies to onboard agents into the environment.

eval-driven-development (benchmark and verification loop)

The team used Terminal Bench scoring, verification runs, self-verification prompts, pre-completion checks, and trace-derived failure analysis to drive each harness iteration.

observability-tracing (agent failure analysis)

Every action and metric is stored in LangSmith, then trace-analysis agents mine those traces for recurring failure modes and harness changes.

tool-use-orchestration (sandboxed coding execution)

Harbor, Daytona, deepagents-cli, middleware, bash tool discovery, and verification/scoring are orchestrated around the coding agent so it can execute tasks autonomously.

multi-agent-orchestration (trace-analysis workflow)

The trace analyzer skill spawns parallel error-analysis agents and uses a main agent to synthesize findings and suggestions.

human-in-the-loop-gate (generalization review)

Human review is kept in the trace-improvement loop to verify proposed changes and prevent overfitting-induced regressions.

model-routing (reasoning compute allocation)

The harness varies reasoning budget across planning, implementation, and verification and discusses future multi-model handoffs from large planning models to smaller implementation models.

Tradeoffs

Optimize the harness, not the model.

Chose System prompts, tools, middleware, tracing, and execution flow changes over Changing or fine-tuning gpt-5.2-codex.

Rationale: The benchmark improvement isolates the value of systems engineering around a fixed model.

Narrow the initial harness optimization surface.

Chose System prompt, tools, and middleware over All possible harness knobs at once.

Rationale: A compressed design space made the experiment tractable while still improving benchmark performance significantly.

Force self-verification before exit.

Chose PreCompletionChecklistMiddleware plus explicit build-verify-fix guidance over Trusting the agent's natural stopping behavior.

Rationale: The most common failure was premature exit after superficial code review, and tests provide a feedback signal for hill climbing.

Inject context instead of relying on search.

Chose LocalContextMiddleware and deterministic environment onboarding over Agent-led discovery of directories, tools, and constraints.

Rationale: Search and context discovery are error-prone, and explicit context lowers avoidable planning failures.

Spend reasoning selectively.

Chose xhigh-high-xhigh reasoning sandwich over xhigh reasoning for every phase.

Rationale: Planning and verification benefit from deeper reasoning, while maximum reasoning everywhere causes timeouts and worse score.

Keep a human check in the trace-improvement loop.

Chose Human verification of proposed harness changes over Fully automated trace-analyzer-driven harness updates.

Rationale: Human review reduces the risk that suggested changes overfit individual tasks and harm generalization.

Gotchas (what bit them)

Agents may stop after superficial self-review.

Root cause: The agent wrote a solution, re-read its own code, decided it looked okay, and exited without running meaningful verification.

Resolution: Add explicit build-verify-fix guidance and a PreCompletionChecklistMiddleware that forces a final verification pass against the task spec.

Autonomous environment discovery is a reliability risk.

Root cause: Terminal Bench environments contain directories, tools, and constraints that agents may discover poorly or too late.

Resolution: Inject directory context, tool availability, testing expectations, file-path requirements, edge-case guidance, and time-budget warnings.

Agents can get stuck in doom loops.

Root cause: After committing to a flawed plan, agents repeat small edits to the same file rather than reconsidering the approach.

Resolution: Track per-file edit counts with LoopDetectionMiddleware and inject reconsideration context after repeated edits.

More reasoning is not always better.

Root cause: Maximum reasoning can burn time and tokens under benchmark timeouts, preventing task completion.

Resolution: Use a reasoning sandwich that spends more reasoning on planning and verification while using high rather than xhigh reasoning for implementation.

Automated harness optimization can overfit.

Root cause: Trace-derived suggestions can improve specific failed tasks while creating regressions elsewhere.

Resolution: Keep human verification in the improvement loop before accepting targeted harness changes.

Harnesses do not transfer perfectly across models.

Root cause: Different models require different prompting and harness iterations, even when general principles transfer.

Resolution: Run several rounds of harness iteration for each task/model pair and track which changes generalize.

Evaluation

LangChain evaluated harness changes by running deepagents-cli on Terminal Bench 2.0 through Harbor/Daytona, storing every action in LangSmith, analyzing trace failure modes, and comparing benchmark scores, timeouts, reasoning-budget settings, token counts, latency, cost, and cross-model results.

Metrics: Terminal Bench 2.0 score improved from 52.8 to 66.5 · absolute improvement was 13.7 points · ranking moved from just outside Top 30 to Top 5 · Terminal Bench 2.0 contains 89 tasks · xhigh-only reasoning scored 53.9% · high reasoning scored 63.6% · Claude Opus 4.6 scored 59.6% with an earlier harness version · more reasoning can burn over 2x more tokens/time · doom loops repeated small variations 10+ times in some traces

Outcomes

• deepagents-cli improved from 52.8 to 66.5 on Terminal Bench 2.0 using harness changes alone.
• The result moved the coding agent from just outside the Top 30 to Top 5 on the leaderboard.
• Trace analysis became a repeatable agent skill for identifying failures and suggesting harness improvements.
• The improved harness added build-verify-fix prompting, local context injection, pre-completion verification, loop detection, and reasoning-budget allocation.
• LangChain published a trace dataset and continues work on multi-model harnesses, memory primitives, and trace-mining methods.

The case reports 89 benchmark tasks, every agent action stored with latency/token/cost metrics, a 13.7-point score gain, 10+ repeated edits in some doom-loop traces, reasoning settings from low to xhigh, and open trace data for the community.