Building and Optimizing AI Programming Agents with MLOps Infrastructure at Scale

weights_&_biases · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Programming-agent optimization

Weights & Biases needed to show how MLOps infrastructure can systematically improve programming agents, moving from a small Tiny Agent harness to top SWEBench performance and toward reinforcement-learning-driven researcher agents.

Constraints:

• Programming agents fail through reasoning mistakes and harness limitations.
• Agent changes need large-scale experiment tracking and reproducible evaluation.
• SWEBench requires running many examples in parallel.
• Future agent improvement may require reinforcement learning over tasks, not only prompt tweaks.

Architecture

The presenter built Tiny Agent as a simple command-line harness with read, write, and run tools around an LLM, then optimized the harness through experiments tracked in Weave. W&B Launch runs hundreds of parallel eval examples, Weave stores traces/results and leaderboards, and later reinforcement-learning tooling is proposed for directly improving agents and researcher-agent workflows.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  llm[LLM] --> agent-harness[Agent Harness]
  agent-harness --> tool-set[Tool Set]
  agent-harness --> weave[Weave]
  launch[Launch] --> swebench[SWEBench]
  launch --> weave[Weave]
  swebench --> leaderboard[Leaderboard]
  weave --> leaderboard[Leaderboard]
  rl-trainer[RL Trainer] --> agent-harness[Agent Harness]
  weave --> rl-trainer[RL Trainer]

Components

Data flow

An agent harness formats memory, tools, and instructions for the LLM, executes read/write/run-command actions, records traces and results in Weave, launches many eval examples through W&B Launch, compares results on leaderboards, and uses identified failures for further harness or RL improvements.

Why this architecture

The talk separates the model from the harness so teams can improve prompts, tools, memory, and model choices independently. Weave and Launch provide the experiment and scale infrastructure needed to iterate across many SWEBench examples rather than judge agent changes anecdotally.

Models used: OpenAI 01 · O1 reasoning model · R1 · Tiny Agent · LLMs

Scale

• Tiny Agent was about 400 lines of code.
  “agent called Tiny Agent. It was a command line. It was command line based in about 400 lines of code and it really worked even back then. And I was like, "Wow, this is incredible." a small wpper totally general purpose tool that I can talk to and get to do for me. And it”transcript, stated
• The presenter topped the SWEBench public leaderboard on January 17, 2025.
  “And on January 17th, 2025, I'm happy to report that I did it. Thank you. My submission used OpenAI's 01 model and held the top position on the public leaderboard for two months.”transcript, stated
• The submission held the top public leaderboard position for two months.
  “My submission used OpenAI's 01 model and held the top position on the public leaderboard for two months. So, how did I do it? The answer, deadlines. My son was born the next day, so I knew I needed to get that result. I grinded it out.”transcript, stated
• Launch enabled hundreds of parallel examples.
  “So with this I could happily run hundreds of parallel uh examples. And this is how I did it for all of my experiments. But there are some problems with this. This is the first problem. Uh running longunning processes on your”transcript, stated

Alternatives considered (roads not taken)

Only change the model.

Rejected because: The presenter held OpenAI 01 fixed and improved the harness through experiments.

Manual local evaluation.

Rejected because: SWEBench optimization needed hundreds of parallel examples and shared experiment tracking.

Stop at maxing out SWEBench.

Rejected because: The talk argues real-world researcher agents require broader tasks and RL beyond benchmark maximization.

Patterns

eval-driven-development (agent optimization)

W&B used SWEBench, Weave traces, Launch, and leaderboards to drive agent improvements.

tool-use-orchestration (programming harness)

Tiny Agent's harness gives the LLM read, write, and run-command tools.

observability-tracing (MLOps)

Weave stores traces and results so failures can be analyzed and compared.

batch-offline-processing (evaluation scale)

Launch runs hundreds of benchmark examples in parallel.

fine-tuning-adaptation (reinforcement learning)

The talk proposes reinforcement learning to directly optimize agent behavior from eval signals.

Tradeoffs

Optimization target

Chose Harness experiments with a fixed model over Only swapping models.

Rationale: Holding the model fixed isolated harness improvements and made progress measurable.

Agent simplicity

Chose Tiny Agent with three tools over A complex initial framework.

Rationale: Read/write/run-command was enough to build a working programming agent and expose the key optimization loop.

Improvement mechanism

Chose Reinforcement learning as a future optimization path over Manual harness changes only.

Rationale: RL can directly train models from eval signals when outcome correctness can be measured.

Gotchas (what bit them)

Programming agents fail from reasoning mistakes, not just missing tools.

Root cause: Earlier agents made logical errors in how they processed data.

Resolution: Use reasoning models and systematic evals to identify and reduce those errors.

Agent harness improvements need experiment infrastructure.

Root cause: Prompt, memory, and tool changes create many variants that must be tracked.

Resolution: Use Weave as the experiment database and Launch for distributed evals.

Benchmark maximization is not enough for real researcher agents.

Root cause: SWEBench is useful but narrower than open-ended research work.

Resolution: Extend tooling toward researcher agents, richer evals, and RL workflows.

Evaluation

The case evaluates programming agents on SWEBench using Weave for traces/results, Launch for distributed execution, and leaderboards for comparing models and harnesses.

Metrics: SWEBench leaderboard position · Parallel eval examples · Trace-level failures · Harness variants · Model comparisons

Outcomes

• Tiny Agent demonstrated a small harness could solve real programming tasks.
• The OpenAI 01 submission topped the SWEBench public leaderboard for two months.
• W&B positioned Weave and Launch as infrastructure for evals, researcher agents, and RL optimization.

The source reports a 400-line Tiny Agent, a January 17, 2025 top SWEBench result, two months at the top, and hundreds of parallel eval examples.