Building and Scaling an AI Coding Agent Through Rapid Iteration and User Feedback

anthropic · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI coding agents

Anthropic needed Claude Code to handle complex, multi-step software engineering work while remaining simple to start, deeply customizable across heterogeneous developer environments, and fast enough to evolve with model capabilities and user feedback.

Constraints:

• the terminal interface gives power users access to existing developer tools but limits visible onboarding and UI affordances
• developer environments vary widely, so the agent must be extensible through hooks, commands, memory files, and sub-agents
• multi-step tasks can stall when the model loses track of pending work
• formal end-to-end coding benchmarks are hard to diagnose when scores change
• model capabilities change quickly, making long-range product planning unreliable
• internal and enterprise feedback arrive at high volume and must be prioritized without over-planning

Architecture

Claude Code is a terminal-first coding agent built around a simple CLI surface, broad access to existing command-line tools, and extensibility primitives such as hooks, custom slash commands, sub-agents, plan mode, to-do lists, and layered Claude.md memory files. Anthropic operates it through rapid internal dogfooding: engineers prototype features, ship them to roughly hundreds to a thousand internal employees, read negative feedback, iterate two or three times, and fast-track features that users love. Evaluation combines end-to-end SWE-bench-style harness tests, triggering evals for tool-use decisions such as web search, and continuous user feedback from internal Slack, GitHub issues, enterprise adopters, designers, PMs, and developers.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user[User] --> terminal-cli[Terminal CLI]
  terminal-cli --> coding-agent[Coding agent]
  coding-agent --> developer-tools[Developer tools]
  coding-agent --> claude-md-memory[Claude.md memory]
  coding-agent --> to-do-list[To-do list]
  coding-agent --> plan-mode[Plan mode]
  coding-agent --> hooks[Hooks]
  hooks --> external-notifications[External notifications]
  coding-agent --> sub-agents[Sub-agents]
  product-engineers[Product engineers] --> dogfood-community[Dogfood community]
  dogfood-community --> feedback-channels[Feedback channels]
  feedback-channels --> product-engineers
  product-engineers --> evaluation-suites[Evaluation suites]
  evaluation-suites --> coding-agent
  coding-agent --> claude-code-sdk[Claude Code SDK]
  claude-code-sdk --> external-agent-builders[External agent builders]

Components

Data flow

A user starts Claude Code in a terminal and gives a coding or product task. The agent receives repository and personal context from Claude.md, can call existing CLI tools, can use custom slash commands, hooks, and sub-agents, and may create a to-do list or plan before editing. New features are prototyped by product engineers, dogfooded internally, and evaluated by user reaction, issue volume, and formal evals. Feedback from Slack, GitHub issues, early enterprise adopters, and internal users feeds back into feature prioritization, where strong convergence pushes features through two or three internal iterations and then into public release.

Why this architecture

Anthropic chose a terminal-first, extensible architecture because the terminal gives an agent immediate access to nearly everything a developer can do, while the sparse interface forces disciplined feature selection. Extensibility through hooks, commands, memory, and sub-agents lets power users adapt the same simple core to heterogeneous environments. The operating model mirrors the product: prototypes beat long docs, internal dogfooding exposes whether users understand or love a feature, and evals plus user feedback catch regressions that benchmark numbers alone cannot explain.

Models used: Claude · Claude Code · Claude Code SDK

Scale

• The internal dogfood community reached hundreds and possibly about a thousand employees.
  “shipping it to dog fooding, which is so that hundreds at this point maybe like a thousand internal employees can try it out.”transcript, stated
• Features often go through two or three internal iterations before public release.
  “a lot of times we've gone through two or three iterations of it internally before we made it public.”transcript, stated
• The internal feedback channel produced new feedback about every 10 minutes.
  “we get a new piece of feedback every I want to say like 10 minutes or so. We love negative feedback.”transcript, stated
• Enterprise-priority fixes typically turned around in one to two weeks.
  “Beyond internal feedback, the team works closely with about 10 early enterprise adopters who are encouraged to be "as loud as possible" about issues, with the team committing to fast service level agreements—typically turning around prioritized fixes within one to two weeks.”summary, stated
• The to-do list problem involved refactors with 20 to 30 changes where the agent might complete only the first five.
  “users employed Claude Code for large refactoring tasks involving 20-30 changes, but the agent would sometimes complete only the first five changes before stopping prematurely.”summary, stated
• The team finds it hard to predict more than six months ahead because model capabilities change quickly.
  “because the models are changing so quickly, we think that it's really hard to predict more than 6 months in the future.”transcript, stated

Alternatives considered (roads not taken)

A richer GUI-first onboarding surface

Rejected because: The team preferred the terminal's immediate access to developer tools and a minimalist interface that avoids overwhelming users.

Formal PRDs for every feature

Rejected because: For smaller features, the hard part was discovering the form factor and prompt behavior through prototypes rather than solving an integration spec.

Teaching users to request planning only in natural language

Rejected because: After one or two months of requests, users still wanted an explicit plan-mode shortcut.

Always trigger web search for questions

Rejected because: The team wants web search to trigger only when freshness or external information is needed.

Multi-page documentation before prototypes

Rejected because: Claude Code makes it cheap to demo and feel the feature before writing a long specification.

Patterns

tool-use-orchestration (agent-runtime)

Claude Code operates through the terminal, letting the agent use existing developer CLIs and project tools instead of requiring each integration to be reimplemented.

agent-memory (context)

Claude.md files provide repo, project, personal, and global context that is included when a Claude Code instance starts.

multi-agent-orchestration (agent-architecture)

Sub-agents are one of the customization primitives used to adapt Claude Code to different developer environments and workflows.

eval-driven-development (quality)

The team uses SWE-bench-style end-to-end evals for harness/model changes and triggering evals for tool-use decisions such as web search.

human-in-the-loop-gate (workflow)

Plan mode and persistent to-do lists give users checkpoints to review intent, progress, and task scope before or during code changes.

data-quality-loop (feedback)

Internal dogfooding, negative feedback, GitHub issues, and enterprise feedback drive which features are iterated, shipped, or dropped.

Tradeoffs

Primary product surface

Chose minimal terminal-first CLI over a heavier web-app-style interface with explicit buttons and onboarding.

Rationale: The terminal gives broad tool access and keeps the interface simple, but it forces brutal prioritization of what appears on screen.

Feature-development process

Chose engineer prototypes plus dogfooding over spec-heavy product planning for every feature.

Rationale: For small agent-product features, the decisive question is whether users understand, trust, and love the behavior, which prototypes reveal faster than docs.

Planning interface

Chose explicit plan mode over only natural-language requests like 'tell me what you will do first'.

Rationale: Although the team wanted users to ask naturally, enough users wanted a clear shortcut that the team added one.

Customization mechanism

Chose hooks and user-configured integrations over only built-in notification surfaces.

Rationale: Users wanted notifications when long-running agent work needed them, and hooks let each environment choose Slack, text, or other integrations.

Roadmap horizon

Chose few-month planning around next-generation model capabilities over a fixed one-to-two-year product vision.

Rationale: Model capabilities change quickly enough that the team does not trust long-range feature prediction.

Gotchas (what bit them)

End-to-end coding evals do not explain themselves.

Root cause: When SWE-bench scores move, the benchmark does not say which harness behavior changed or why.

Resolution: Read transcripts for themes and pair end-to-end evals with more targeted triggering evals where possible.

Agents may create planning overhead for trivial tasks.

Root cause: The to-do-list trigger can fire even when there is only one task to track.

Resolution: Add triggering evals with trajectories where to-do lists should not be used or should contain an appropriate number of items.

Models are too eager to code even when users want planning.

Root cause: Current models often push toward implementation after presenting a plan.

Resolution: Expose explicit plan mode for now, while hoping future models follow planning-vs-execution directions better.

Terminal-first power limits accessibility for non-technical users.

Root cause: The terminal interface is hard to explain to users who have never used it before.

Resolution: Keep professional developers as the core market while exploring form factors for data science, PM, design, marketing, and sales.

Capability failures are hard to diagnose.

Root cause: A model failure might be caused by wrong context, wrong model choice, or a fundamental capability gap.

Resolution: Build product intuition by pushing models and judging whether a capability is 80% present or only 10% present.

Evaluation

Claude Code evaluation combines imperfect end-to-end harness evals such as SWE-bench, triggering evals for tool-use decisions such as web search or to-do-list creation, and continuous feedback from internal users, GitHub issues, enterprise adopters, and product critique.

Metrics: SWE-bench score changes · tool-triggering correctness · web search trigger rate · to-do list trigger appropriateness · GitHub issue votes · internal feedback frequency · enterprise customer requests

Outcomes

• Claude Code spread virally inside Anthropic before external launch
• the product expanded beyond developers into research, data science, product management, and product design workflows
• features could move from prototype to public release after two or three internal iterations
• PMs and designers began contributing code directly for smaller changes
• the team identified CLI leadership, SDK growth, and broader accessibility as near-term strategic pillars

The case describes roughly hundreds to a thousand internal dogfood users, feedback about every 10 minutes, and prioritized fixes often within one to two weeks.