Building and Scaling Production Code Agents: Lessons from Replit

replit · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production code agents

Replit needed to ship a fully automated code agent for users building apps from natural language, while optimizing for the right user persona, reducing time-to-first-working-app, detecting strange long-tail agent failures, building rigorous product-specific evals, and upskilling engineers to design reliable agent tools.

Constraints:

• public benchmarks such as SWE-bench did not match Replit's interactive AI-first-coder use case
• interactive users need fast feedback loops rather than expensive long-running search
• agents fail in long-tail ways that ordinary crash monitoring and user-session metrics do not reveal
• manual trace review is impossible at thousands of traces per day
• evaluation harnesses are resource intensive but needed to detect regressions and model changes
• engineers needed intuition for LLM/tool design, which differs from ordinary API design

Architecture

Replit built a production code agent inside its cloud development environment. The agent receives a user's natural-language app request, rewrites it to clarify intent and tech-stack assumptions, can enter rapid build mode to generate 10-15 cohesive files quickly, then falls back to a traditional agentic loop to fix issues. The agent sees views similar to the user, including open files through an LSP and real-time console logs, and can interactively iterate with the user. Replit monitors traces in LangSmith, detects failures through rollback logs, rollback-frequency hotspots, negative sentiment, support/social reports, and evaluates behavior with a homegrown Python harness that spins up Chrome instances and Docker containers, using Anthropic computer use as a web agent to test generated apps in parallel. Engineers contribute product-specific evals and include example traces on agent PRs to build intuition.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  replit-workspace[Replit workspace] --> replit-agent[Replit Agent]
  replit-workspace --> agent-views[Agent views]
  replit-agent --> prompt-rewriter[Prompt rewriter]
  prompt-rewriter --> rapid-build-mode[Rapid build mode]
  rapid-build-mode --> agentic-repair-loop[Agentic repair loop]
  prompt-rewriter --> agentic-repair-loop
  agentic-repair-loop --> agent-views
  replit-agent --> rollback-failure-detector[Rollback failure detector]
  rollback-failure-detector --> sentiment-and-feedback-signals[Sentiment and feedback signals]
  sentiment-and-feedback-signals --> langsmith-trace-monitoring[LangSmith trace monitoring]
  replit-agent --> langsmith-trace-monitoring
  replit-agent --> evaluation-harness[Evaluation harness]
  evaluation-harness --> langsmith-trace-monitoring
  evaluation-harness --> evaluation-pool[Evaluation pool]
  evaluation-pool --> engineer-upskilling-loop[Engineer upskilling loop]

Components

Data flow

A user describes an app or change in Replit Agent. The request is rewritten to clarify intent and expose assumptions, such as whether to use Flask or Express. For new apps, rapid build mode emits many starter files from templates and custom prompting, then the ordinary agentic loop repairs issues. The agent uses IDE-like views, including files through LSP and console logs, to inspect the environment. User rollbacks, negative language, feedback, social/support reports, and LangSmith traces identify failures. Evaluation jobs spin up test environments in Chrome and Docker, let an Anthropic computer-use web agent behave like a user, log results, and feed new failures into a growing eval pool owned by the AI team but extendable by engineers across the company.

Why this architecture

Replit changed the architecture around the target persona. SWE-bench-style GitHub-issue-to-PR agents and Monte Carlo Tree Search can work for asynchronous engineering-manager workflows, but Replit's AI-first coders want tight loops and fast app starts. Rapid build mode trades initial precision for speed, then the agentic loop repairs small issues. Rollbacks and sentiment are embedded into the product because generic metrics and crash monitoring cannot distinguish satisfied exits from frustration or detect loops. The eval harness is treated like integration-test infrastructure: expensive to build, but cheap to extend and essential for model changes, regressions, and engineer learning.

Models used: hosted LLMs from labs · self-hosted code completion model · self-hosted code repair model · GPT-4o · GPT-4 mini · Claude 3.5 Haiku · Anthropic computer use · web agent

Scale

• Replit had about 30 million signed-up users.
  “we've had a lot of success we've got about 30 million users uh signed up”transcript, stated
• Replit deployed the agent in September.
  “in September uh we kind of deployed uh the repet agent”transcript, stated
• Monte Carlo Tree Search with five parallel agents costs about 5x as much and slows the loop.
  “if you run you know five agents in parallel you pay five times as much um and it also kind of slows things down”transcript, stated
• The original method could take 6-7 minutes to produce a working app, while rapid build mode reduced it to under 2 minutes.
  “this ultimately took us from a working application taking six or seven minutes to get started to being under 2 minutes”transcript, stated
• The fast-start path can emit 10-15 files, whereas the methodical loop created 5-10 files with 20-30 second loops.
  “when you needed to create say you know five or 10 different files to get started um and each Loop kind of took like 20 30 seconds”transcript, stated

  “the agent can very easily actually dump out 10 15 files”transcript, stated
• Replit generates thousands and thousands of traces every day.
  “we generate thousands and thousands of traces every day and it's simply too much for us to actually read through”transcript, stated
• A 5% failure rate requires about 20 runs to expect to see a failure.
  “if something fails 5% of the time you need to run it 20 times before you expect to actually see a failure”transcript, stated
• The AI team grew from around 8 engineers, with an initial 3-engineer agent prototype, to roughly 20 engineers around the agent effort.
  3 supporting quotes

  “repet AI team uh about 12 months ago was around eight Engineers”transcript, stated

  “the kind of initial prototype for the agent was really like a three Engineers”transcript, stated

  “those three Engineers working on the agent expanded massively to like 20”transcript, stated

Alternatives considered (roads not taken)

Optimize primarily for SWE-bench

Rejected because: SWE-bench measured GitHub issue-to-PR workflows, while Replit users wanted to build ideas from scratch in a tight loop.

Monte Carlo Tree Search for interactive users

Rejected because: It improves long-trajectory accuracy but costs five times as much and slows the loop, which hurts AI-first coders and traditional engineers.

Traditional methodical agentic loop for app bootstrap

Rejected because: It was too slow for getting-started flows, taking several minutes before users saw something usable.

Explicit feedback buttons as the main failure signal

Rejected because: Users rarely used them and felt like they were screaming into the void; rollbacks gave immediate value and clearer signal.

Public evaluations as the main quality bar

Rejected because: Public evals measure important generic skills but not Replit's specific marketing-site and app-building workflows.

Patterns

query-rewriting (intent)

Replit rewrites user prompts before coding to expand details, clarify intent, and expose tech-stack choices early.

prompt-chaining (workflow)

Rapid build mode generates an initial cohesive application skeleton, then the ordinary agentic loop repairs small issues.

observability-tracing (operations)

Replit uses LangSmith traces and application-level rollback/sentiment signals to find agent loops, bypassed blocklists, and frustration cases that crash metrics miss.

data-quality-loop (feedback)

Rollbacks, negative sentiment, support, and social reports become signals for engineers to inspect traces and add new evaluations.

eval-driven-development (quality)

Replit treats evaluations as a long-term investment, using a homegrown harness, Braintrust for some evals, LangSmith traceability, Chrome/Docker test environments, and Anthropic computer use.

model-routing (model-ops)

Replit distinguishes hosted lab models, self-hosted smaller models, code completion, code repair, and future small-model/MCTS options by task, latency, and cost.

cost-aware-routing (cost)

The team avoids expensive parallel-agent methods for interactive personas because cost and latency outweigh the accuracy gains.

tool-use-orchestration (agent-runtime)

The agent uses user-visible views such as files, LSP diagnostics, and console logs as context for coding and debugging.

Tradeoffs

Target persona

Chose AI-first coder and interactive getting-started loop over engineering-manager asynchronous issue-to-PR workflow.

Rationale: The personas value different speeds, budgets, codebase sizes, and interaction styles, and improving one can harm another.

Startup generation path

Chose rapid build mode then repair over methodical loop from the first file.

Rationale: Users needed a working app quickly; initial bulk generation could be imperfect because the loop could fix small issues afterward.

Failure signal

Chose rollback monitoring over feedback button as the main signal.

Rationale: Rollbacks give users immediate value and create a clear mistake signal, while feedback buttons were rarely used.

Evaluation investment

Chose homegrown product-specific eval harness over public benchmarks alone.

Rationale: Replit needed tests that matched its app-building workflows and could grow with new product failures.

Organizational speed

Chose slow down to build the eval harness over continue vibes-based first-to-market shipping.

Rationale: Senior engineers argued the harness would let the organization move faster later, like integration tests.

Gotchas (what bit them)

Optimizing the wrong benchmark produced the wrong product behavior.

Root cause: SWE-bench measured GitHub issue-to-PR workflows, while Replit's users wanted to build new ideas from scratch quickly.

Resolution: Define target personas and optimize the agent around tight-loop AI-first coder workflows.

The first methodical agent loop felt slow for starting a new app.

Root cause: Creating several files through 20-30 second loops delayed the first usable application.

Resolution: Add rapid build mode to emit a cohesive starter app, then switch back to the agentic loop for repairs.

Agents fail in strange ways that ordinary monitoring does not catch.

Root cause: Agents can loop, wander down tangents, bypass blocklists, or frustrate users without crashing the app.

Resolution: Embed failure detection in the product through rollbacks, sentiment, traces, social reports, and support traces.

Manual trace review does not scale.

Root cause: The system generates thousands of traces per day.

Resolution: Use traces for drilldown, but rely on rollbacks, sentiment, support, and social signals to select traces worth investigating.

Vibes-based development produced regression-prone prompt hacks.

Root cause: Playing with the agent and making changes that feel better does not quantify failure rates or catch regressions.

Resolution: Invest in product-specific evals that behave like integration tests and grow as issues are found.

Agent tool design is not the same as API design.

Root cause: More fields and tool affordances can push the model out of distribution and cause behavior to go off the rails.

Resolution: Use eval harnesses, hands-on LLM time, and example traces on every agent PR to build engineer intuition.

Evaluation

Replit evaluates the agent with product-specific internal evals, a homegrown Python harness, Braintrust for some traditional chat evaluations, LangSmith traceability, Anthropic computer use as a web agent, Chrome instances, Docker containers, parallel app testing, rollback frequency, negative sentiment, social/support reports, example traces on PRs, and reactive tests added whenever new failures appear.

Metrics: SWE-bench score · time to working application · rollback count and rollback point frequency · negative sentiment in user input · trace volume · failure rate · success rate · public benchmark performance · internal product-specific eval pass rate · new-model eval results · example traces per PR · parallel web-agent test outcomes

Outcomes

• Replit deployed a fully automated code agent in September
• rapid build mode reduced time-to-working-application from 6-7 minutes to under 2 minutes
• rollback monitoring became a high-signal way to find agent mistakes
• the team built a homegrown eval harness with Chrome, Docker, and Anthropic computer use
• internal evaluations became a moat, a safety net for large changes, and a way to test new models before release
• the agent effort grew from a 3-engineer prototype into roughly 20 engineers across agent and platform integrations
• example traces on every agent PR helped engineers learn how agents use tools
• agent views include LSP-fed files and real-time console logs rather than traditional breakpoint debugging

Replit's production context included roughly 30 million signed-up users, thousands of traces per day, a fast-start target under two minutes, and an evaluation discipline designed to detect changes from 50% to 90-99% success over many runs.