Evolution of AI Agents: From Manual Workflows to End-to-End Training

openai · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: agent model training and productized agent systems

OpenAI's agents team describes the shift from human-designed LLM workflows toward end-to-end trained agents because multi-step workflows suffer compounding errors and oversimplify messy real-world task processes.

Constraints:

• Traditional LLMs were not trained for agentic work, so errors compounded across trajectories.
• Human-designed workflows can oversimplify how real experts solve messy tasks.
• Agentic products must interact with real-world tools such as browsers, codebases, and file systems.
• Deep Research quality depends on up-front question framing and follow-up clarification.
• Agentic coding tools need context discovery, sandboxing, customization, and user trust before larger delegation.

Architecture

OpenAI's direction is to train models end-to-end with reinforcement learning on complete workflow-like tasks, then expose those capabilities through products such as Deep Research, Operator, and Codex CLI. The architecture increasingly moves task decomposition and recovery into the trained model, while product surfaces add clarification loops, browser or filesystem tools, sandboxing, site-specific instructions, repository customization, and open-source community feedback.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  end-to-end-trained-model["End-to-end trained model"] --> deep-research["Deep Research"]
  deep-research --> clarification-loop["Clarification loop"]
  clarification-loop --> end-to-end-trained-model
  end-to-end-trained-model --> operator["Operator"]
  operator --> site-specific-instructions["Site-specific instructions"]
  site-specific-instructions --> operator
  end-to-end-trained-model --> codex-cli["Codex CLI"]
  codex-cli --> network-sandbox["Network sandbox"]
  network-sandbox --> codex-cli

Components

Data flow

A user delegates a research, browser, or coding task; the agent uses model reasoning and tools to explore the environment, recover from failed steps, ask clarifying questions when useful, execute actions in a browser or local codebase, and return a report, completed task, or code patch.

Why this architecture

The team argues that training agents on full trajectories is more robust than manually wiring many LLM calls because trained models can see failures, recover, and generalize beyond the workflow structure a human would have guessed in advance.

Models used: Reasoning models · Reinforcement learning trained agents · Deep Research · Operator · Codex CLI · o3 · o4-mini

Scale

• The speaker gives a 10-step workflow example where 90% per-step accuracy can still fall off badly.
  “even if you're, you know, 90% accurate on one step, if you have to take 10 steps, um, then your accuracy will fall off.”transcript, stated
• Deep Research can spend 10 or 20 minutes generating a research response.
  “then they'll put that into deep research and have it go, you know, spend uh 10 or 20 minutes coming up with a much more uh research response.”transcript, stated
• Codex CLI had more than 20,000 GitHub stars and about 100 contributors in the talk.
  “there's like more than 20,000 stars on GitHub, which is amazing to me. Um we've got like around 100 people have contributed to it.”transcript, stated

Alternatives considered (roads not taken)

Human-designed multi-agent workflows.

Rejected because: They suffer compounding errors and may oversimplify how real expert workflows proceed.

Single-shot answers instead of Deep Research.

Rejected because: Some tasks need a much more thorough answer and benefit from clarification before research begins.

IDE-only AI coding assistance.

Rejected because: The CLI agent is positioned for delegated de novo or separable chunks of work rather than inline suggestions only.

Patterns

eval-driven-development (RL trajectory training)

Agent models are trained against complete task success so they encounter and learn recovery from failures during training.

tool-use-orchestration (agent tools)

Operator uses a browser, while Codex CLI uses filesystem and command-line tools to complete delegated tasks.

human-in-the-loop-gate (research clarification)

Deep Research asks follow-up questions and can return interim findings for user feedback during long-running work.

context-compaction (codebase exploration)

Codex CLI is described as contextless at task start, forcing it to explore only the files and information it needs.

Tradeoffs

Train agents end-to-end.

Chose RL-trained trajectory models over Manual multi-agent workflows around API calls.

Rationale: End-to-end training lets the model learn recovery and task-specific process control instead of relying on brittle human decomposition.

Launch Operator as an early product surface.

Chose Technology preview with power users over Waiting until every user could rely on it daily.

Rationale: The product gives users value and a feel for the direction even though it is not yet for everyone.

Run Codex CLI locally with sandboxing.

Chose Local code execution with network sandbox over Unrestricted remote automation.

Rationale: The agent can work on the user's laptop while limiting network access for safety.

Gotchas (what bit them)

Small step errors compound across agent trajectories.

Root cause: Most LLMs were historically not trained for full agentic workflows.

Resolution: Train agents end-to-end so they encounter failures and learn recovery strategies.

Human-specified workflows can be too simple for real-world tasks.

Root cause: The real world is messy and expert processes do not always fit a planned decomposition.

Resolution: Let models learn complete processes through reward and trajectories instead of hand-designing every step.

Deep Research output quality depends on framing.

Root cause: The model follows the initial research framing closely, so vague questions can produce less useful reports.

Resolution: Use follow-up questions or a smaller model to help flesh out the research question before the run.

Evaluation

The talk frames evaluation as full-task reinforcement learning, product usage feedback, user interaction observation, and practical success of products such as Deep Research, Operator, and Codex CLI.

Metrics: Complete task success · Failure recovery during trajectories · User value and adoption · GitHub stars and contributors · Research result quality · Repeatability of site-specific instructions

Outcomes

• Deep Research, Operator, and Codex CLI are presented as production agentic products.
• End-to-end trained reasoning models can recover from wrong search terms and other multi-step failures.
• Codex CLI demonstrated strong open-source uptake and internal use.
• Site-specific instructions make Operator more repeatable and faster.

The talk cites products used for business, research, scientific, travel, shopping, and programming workflows; Codex CLI had more than 20,000 GitHub stars and around 100 contributors.