Engineering and Optimizing an Agent Harness for Production AI Coding Assistants

cursor · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI coding-agent harness engineering

Cursor needed a production agent harness that could keep coding assistants effective as models, context strategies, tools, provider behaviors, and real-user workloads changed.

Constraints:

The context window must carry prompts, tool descriptions, conversation state, and user requests without accumulating stale or harmful context.
Harness quality cannot be measured by offline benchmarks alone.
Tool-call errors can poison context and derail coding sessions.
Different models prefer different tool formats, prompts, and behavior assumptions.
Model switching mid-conversation creates cache, tool-shape, and out-of-distribution history problems.

Architecture

Cursor treats the agent harness as a continuously evaluated software product. It moved from static, guardrail-heavy context toward dynamic context retrieval, measures harness changes through CursorBench plus online A/B tests, tracks Keep Rate and user-response sentiment, classifies tool errors, uses anomaly alerts and Cloud Agents for repair, and customizes prompts/tools per model and model version.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  coding-user-request["Coding user request"] --> model-specific-agent-harness["Model-specific agent harness"]
  model-specific-agent-harness --> dynamic-context-retrieval["Dynamic context retrieval"]
  model-specific-agent-harness --> coding-tools["Coding tools"]
  dynamic-context-retrieval --> coding-tools
  coding-tools --> tool-error-telemetry["Tool-error telemetry"]
  cursorbench-and-offline-evals["CursorBench and offline evals"] --> harness-experiment-loop["Harness experiment loop"]
  online-ab-tests["Online AB tests"] --> harness-experiment-loop
  tool-error-telemetry --> harness-experiment-loop
  harness-experiment-loop --> model-specific-agent-harness

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Coding user request`	The coding task and conversation state that drive agent work.	`Model-specific agent harness`
`Model-specific agent harness`	Configures prompts, tools, tool formats, guardrails, and context behavior per model.	`Dynamic context retrieval`, `Coding tools`
`Dynamic context retrieval`	Lets the agent pull files, terminals, past conversations, or tools while it works.	`Coding tools`
`Coding tools`	Patch, search, edit, and environment tools used by the coding agent.	`Tool-error telemetry`
`CursorBench and offline evals`	Standardized offline quality checks for harness and model changes.	`Harness experiment loop`
`Online AB tests`	Deploys harness variants side by side on real usage.	`Harness experiment loop`
`Tool-error telemetry`	Classifies expected and unknown tool failures by cause and model.	`Harness experiment loop`
`Harness experiment loop`	Uses offline evals, online metrics, sentiment, and error analysis to ship harness changes.	`Model-specific agent harness`

Data flow

A coding request enters a model-specific harness with prompts, tool descriptions, conversation state, and available context. The agent dynamically fetches more context and calls tools while metrics and logs capture latency, token efficiency, tool calls, cache hits, code retention, user sentiment, and tool errors; offline and online evaluations feed harness changes.

“the context window starts with the system prompt and tool descriptions, followed by the current state of the conversation, and finally the user's request.”article, stated

“the model can decide when to pull additional information into the context window like past conversations, active terminal sessions, or relevant tools.”article, stated

Why this architecture

Cursor uses a layered evaluation and harness-customization architecture because the harness and model jointly determine agent quality, and neither benchmarks nor simple operational metrics fully capture whether the agent helped the user. The system adapts context and tools as model capabilities improve.

“The harness and the model together determine how good the agent is, but "good" is hard to pin down. To locate it, we've built several layers of measurement.”article, stated

“All of our harness abstractions are model agnostic and can be heavily customized for every model we support.”article, stated

Models used: OpenAI models · Anthropic Claude models · language model sentiment analysis · CursorBench

Scale

The coding agent started in late 2024 and evolved as models improved.
“models were much worse at choosing their own context and we invested lots of context engineering work into creating guardrails”article, stated
Online tests compare two or more harness variants on real usage.
“we deploy two or more harness variants side by side and A/B test them on real usage.”article, stated
A focused sprint drove unexpected tool-call errors down by an order of magnitude.
“Over the course of a focused sprint earlier this year, we drove unexpected tool call errors down by an order of magnitude.”article, stated
The tool reliability chart claims at least two or often three nines of reliability.
“In a focused sprint earlier this year, we drove all tool calls to at least 2 or often 3 9s of reliability.”article, stated

Alternatives considered (roads not taken)

Keep substantial static context in every session.

Rejected because: As models improved, Cursor moved toward leaner static context and more dynamic context retrieval.

“We also provided substantial amounts of static context that was always available to the agent at the start of each session.”article, stated

Rely entirely on public and offline benchmarks.

Rejected because: Benchmarks only approximate real usage, so Cursor also runs online A/B tests and user-derived metrics.

“even the best benchmarks only approximate real usage, meaning we’d miss important signals if we relied on them entirely.”article, stated

Use a single tool interface across models.

Rejected because: Models trained on unfamiliar tool formats spend extra reasoning tokens and make more mistakes.

“Either model could use either tool, but giving it the unfamiliar one costs extra reasoning tokens and produces more mistakes.”article, stated

Switch models mid-conversation without special handling.

Rejected because: Different prompts, tools, histories, and caches make mid-chat switching difficult.

“It’s especially tricky to design the harness to support users switching models mid conversation, because different models have different behaviors, prompts, and tool shapes.”article, stated

Patterns

`tool-use-orchestration` (orchestration)

Cursor provisions prompts and tool formats per model and tracks tool-call errors as a core quality surface.

“in our harness, we provision each model with the tool format it had during training.”article, stated

`eval-driven-development` (evaluation)

Harness changes are assessed with CursorBench, offline evals, online A/B tests, Keep Rate, and model-read user sentiment.

“We measure agent quality in these tests through a variety of metrics. Some are straightforward like latency, token efficiency, tool call count, and cache hit rate.”article, stated

`observability-tracing` (operations)

Cursor classifies tool errors, computes baselines per tool and model, and alerts on regressions.

“We compute baselines per-tool and per-model, because different models may mess up tool calls at different rates.”article, stated

`context-compaction` (context)

For mid-chat model switching, Cursor experimented with summarizing the conversation to reduce provider-cache penalty while acknowledging detail loss risk.

“We have experimented with mitigating this by summarizing the conversation at switch time, which provides the model with a clean summary that reduces the cache penalty.”article, stated

Tradeoffs

Context strategy

Chose Dynamic context that the agent can fetch while working. over Large static context loaded at session start..

Rationale: Improved models can choose context better, so static context and guardrails can be reduced.

“we’ve adapted to increasing model capability by knocking down guardrails and providing more dynamic context, which can be fetched by the agent while it works.”article, stated

Quality measurement

Chose Offline benchmarks plus online user-derived metrics. over Offline benchmarks alone..

Rationale: Real usage reveals signals benchmarks miss, including code retention and user satisfaction.

“The first is the “Keep Rate” of agent-generated code. For a given set of code changes that the agent proposed, we track what fraction of those remain in the user’s codebase”article, stated

Context summarization model

Chose Shelve a more expensive model for context summarization. over Paying for a higher-cost summarization model with negligible quality gains..

Rationale: The online experiment found the quality improvement was not worth the extra cost.

“we tried a more expensive model for context summarization and observed it made a negligible difference in agent quality that wasn’t worth the higher cost.”article, stated

Gotchas (what bit them)

Failed tool calls degrade future reasoning.

Root cause: Errors remain in context and waste tokens, producing context rot.

Resolution: Classify tool errors, alert on regressions, and drive tool reliability improvements.

“errors remain in context, wasting tokens and causing “context rot,” where accumulated mistakes degrade the quality of the model's subsequent decisions.”article, stated

Expected errors can mask real harness bugs.

Root cause: A timeout may come from tool performance, codebase size, or an inefficient model query.

Resolution: Use anomaly detection against per-tool and per-model baselines.

“a grep search timeout might be because of a performance issue with the tool, or the codebase might just be huge and the model formed an inefficient query.”article, stated

Model-specific behavior makes one-size-fits-all harnesses wasteful.

Root cause: Providers and model versions differ in training, tool format, literalness, and tolerance for imprecise instructions.

Resolution: Customize prompts and tools for each provider and model version.

“OpenAI’s models tend to be more literal and precise in their instruction following, whereas Claude is a bit more intuitive and more tolerant to imprecise instructions.”article, stated

Summaries can lose critical details during model switching.

Root cause: A complex task may depend on details that a switch-time summary omits.

Resolution: Generally recommend staying with one model unless there is a reason to switch, or use a fresh-context subagent.

“But if the user is deep into a complex task, the summary can lose important details.”article, stated

Evaluation

Cursor combines public benchmarks, CursorBench, offline evals, online A/B tests, Keep Rate, user-response sentiment, latency, token efficiency, tool-call count, cache hit rate, and tool-error baselines.

Metrics: CursorBench score · Latency · Token efficiency · Tool call count · Cache hit rate · Keep Rate · User satisfaction sentiment · Unknown and expected tool-error rates

“The first is the “Keep Rate” of agent-generated code. For a given set of code changes that the agent proposed, we track what fraction of those remain in the user’s codebase after fixed intervals of time.”article, stated

“we use a language model to read the user's responses to the agent’s initial output in order to capture semantically whether the user was satisfied or not.”article, stated

Outcomes

Moved from heavy static context and guardrails to dynamic context retrieval.
Drove unexpected tool-call errors down by an order of magnitude.
Created layered measurement across offline benchmarks, online experiments, Keep Rate, sentiment, and tool telemetry.
Supported model-specific harness customization and mid-chat model-switching safeguards.

The article cites late-2024 coding-agent origins, weekly automation, A/B tests on real usage, two-to-three-nines tool reliability, and an order-of-magnitude error reduction.

“Over the course of a focused sprint earlier this year, we drove unexpected tool call errors down by an order of magnitude.”article, stated