Building State-of-the-Art AI Programming Agents with OpenAI's o1 Model

weights_&_biases · Tech · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: software engineering agent evaluation

Weights & Biases built an autonomous programming agent for SWE-Bench-Verified, where the system had to repair real GitHub issues inside Docker images by reading code, editing files, running tests, and deciding when a fix was complete.

Constraints:

• The benchmark uses 500 GitHub issues, Docker images, and held-out unit tests rather than toy coding prompts.
• The agent had to operate autonomously in a container, iteratively reading, writing, testing, and refining code.
• Long agent trajectories created context-management pressure, so the system needed memory compression around the primary reasoning model.
• o1 sometimes reasoned incorrectly about event ordering after edit-test-edit sequences, so temporal state could not be left entirely to prompting.
• The result was also a product demonstration by W&B's CTO, so benchmark and tooling claims need to be read with promotional context.

Architecture

The agent used o1 in high-reasoning mode as the main step and editing driver, wrapped by a GPT-4o memory compressor, custom Python code-editing tools, auto-commands after edits, five parallel rollouts per SWE-Bench instance, and an o1 crosscheck step to choose the best rollout. Weave tracked experiments and evaluations, while Eval Studio and Phaseshift emerged as supporting analysis and composition tools.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  swe-bench-verified-instance["SWE-Bench-Verified instance"] --> o1-agent-driver["o1 agent driver"]
  o1-agent-driver --> custom-python-editor-tools["custom Python editor tools"]
  custom-python-editor-tools --> auto-command-runner["auto-command runner"]
  auto-command-runner --> swe-bench-verified-instance
  auto-command-runner --> o1-agent-driver
  o1-agent-driver --> gpt-4o-memory-compressor["GPT-4o memory compressor"]
  gpt-4o-memory-compressor --> o1-agent-driver
  o1-agent-driver --> parallel-rollout-workers["parallel rollout workers"]
  parallel-rollout-workers --> crosscheck-selector["crosscheck selector"]
  crosscheck-selector --> weave-evaluation-and-trace-store["Weave evaluation and trace store"]
  weave-evaluation-and-trace-store --> eval-studio["Eval Studio"]
  weave-evaluation-and-trace-store --> phaseshift["Phaseshift"]

Components

Data flow

A SWE-Bench issue and container entered the agent loop; o1 inspected context, edited code with custom Python tools, auto-commands ran after each edit, GPT-4o compressed long step history, five rollouts explored candidate fixes, and an o1 crosscheck selected the final solution while Weave recorded evaluations and traces.

Why this architecture

The architecture separates high-value reasoning from support tasks: o1 performs coding decisions, GPT-4o compresses history, custom tools reduce context waste, auto-commands compensate for temporal-ordering weakness, and parallel rollouts plus crosscheck spend extra compute where benchmark accuracy matters.

Models used: OpenAI o1 with reasoning_mode high · GPT-4o · o1 tie-breaker

Scale

• The benchmark contains 500 verified GitHub issues.
  “It’s a set of 500 github issues, docker images, and held out unit tests.”article, stated
• The reported system resolved 64.6% of SWE-Bench-Verified issues.
  “I made an OpenAI o1-based AI programming agent that is now state of the art on SWE-Bench-Verified! It resolves 64.6% of issues.”article, stated
• Development involved 977 evaluations over roughly two months.
  “It took tons of iteration and analysis over the last two months to get this agent working so well.”article, stated

  “I ran 977 evals in the course of achieving this solution.”article, stated
• Each problem used five parallel rollouts before crosscheck selection.
  “5 parallel rollouts for each instance, and a final “crosscheck” step for choosing the best rollout, using an o1 tie-breaker”article, stated
• The prompt excerpt shown was 7 of 58 task-instruction lines.
  “That’s 7 out of 58 lines in the full task instructions portion of the prompt.”article, stated

Alternatives considered (roads not taken)

Use OpenAI's basic o1 agent framework.

Rejected because: W&B reports that its richer agent framework substantially outperformed OpenAI's published o1 result.

Prompt o1 to reason through event ordering instead of changing the architecture.

Rejected because: The author considered this plausible but avoided it because such prompts more often triggered invalid prompt detection.

Rely on generic code-manipulation tools.

Rejected because: The team built custom Python editor tools specifically to preserve and use model context efficiently.

Use a single rollout per problem.

Rejected because: The chosen design spends test-time compute on five rollouts and a final selector, implying a quality preference over single-attempt cost/latency.

Patterns

tool-use-orchestration (agent architecture)

The coding agent used custom editor tools and auto-commands as first-class controls around model actions instead of relying on raw text generation.

context-compaction (agent memory)

A GPT-4o memory component compressed the step history so o1 could continue long repair trajectories with reduced context pressure.

eval-driven-development (evaluation)

The team used Weave evaluations and 977 eval runs to iterate on prompt instructions, tool design, and agent behavior.

llm-as-judge (evaluation)

The crosscheck step used o1 as a tie-breaker to choose among multiple rollouts.

prompt-versioning (llmops)

Phaseshift was framed around versioning data and code together so agent changes could be inspected during iteration.

Tradeoffs

Use o1 for core agent reasoning and editing.

Chose A high-reasoning o1 driver over A basic agent framework or weaker prior completion-style models.

Rationale: The author attributes the result partly to o1's bug-finding, instruction-following, and repository reasoning behavior.

Handle temporal-ordering weakness architecturally.

Chose Auto-commands after every edit over Prompt-only temporal reasoning instructions.

Rationale: Prompts might help, but the specific prompt style triggered invalid prompt detection more often, so the tool layer reduced the reasoning burden.

Spend extra compute on multiple rollouts and crosscheck.

Chose Five parallel rollouts plus an o1 tie-breaker over One autonomous attempt per benchmark instance.

Rationale: The article presents crosscheck selection as a quality-improving mechanism, but the cost/latency burden is implied by running multiple attempts.

Build Phaseshift in TypeScript.

Chose TypeScript over An unspecified alternative implementation language.

Rationale: The author says TypeScript's type system helped reason about agent interfaces and composition.

Gotchas (what bit them)

o1 could misread the order of edit and test events.

Root cause: The model did not always reason correctly about temporal ordering in agent trajectories.

Resolution: Move temporal bookkeeping into auto-commands that run after every edit rather than depending only on model reasoning.

More instructions can be risky with older models, even when o1 handled them well here.

Root cause: The author had prior experience that adding prompt detail could degrade adherence elsewhere in the prompt.

Resolution: Treat instruction-heavy prompts as model-specific and validate them through trajectory review and evaluations.

The crosscheck mechanism is promising but under-specified in the public write-up.

Root cause: The author explicitly defers details of how the N-rollout selector works.

Resolution: Use the pattern cautiously until implementation details and contribution versus extra test-time compute are documented.

Evaluation

The work was evaluated on SWE-Bench-Verified and developed through Weave-tracked experiments, trajectory review, live-run analysis, and statistical comparison in Eval Studio.

Metrics: 64.6% resolved on SWE-Bench-Verified · 977 evaluations during development · 5 parallel rollouts per instance · 500 benchmark issues

Outcomes

• The agent reportedly topped the SWE-Bench-Verified leaderboard with a 64.6% resolve rate.
• W&B built or improved supporting tools including Eval Studio, Phaseshift, and Weave playground features.
• The project yielded practical guidance about o1 prompting, context compression, custom tool design, and temporal-ordering failure modes.

The article reports 500 benchmark issues, 5 rollouts per instance, 977 evaluations over roughly two months, and a 64.6% resolved rate.