Building a Production-Ready Multi-Agent Coding Assistant

replit · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production multi-agent coding assistant

Replit needed a coding agent that could build applications from user prompts while limiting long-trajectory agent failures, exposing many developer tools safely, validating progress, debugging production traces, and balancing accuracy, cost, and latency.

Constraints:

• Long autonomous trajectories compound mistakes until the agent crashes or misses the goal
• Too many tools in one agent scope caused errors
• Tool invocation reliability was poor with standard function-calling APIs for complex tools
• Coding-agent behavior requires observability and replay once it reaches production
• End-to-end coding-agent evals were not mature enough to fully gate launch
• The product needed users engaged without asking for feedback too often

Architecture

Replit Agent evolved from a basic ReAct loop into a scoped multi-agent team with a manager, editors, and a verifier. Each sub-agent sees the minimum tool/instruction scope needed. Tools are invoked through generated Python DSL code parsed on Replit's backend rather than raw function-calling APIs. The verifier can inspect the app, run checks, take screenshots, and ask the user when progress stalls. LangSmith traces store full per-step state so production bugs can be pinpointed and replayed. Claude 3.5 Sonnet handles most code generation/editing, with smaller model calls for compression/watchdog work.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user[User] --> replit-agent[Replit Agent]
  replit-agent --> manager-agent[Manager Agent]
  manager-agent --> editor-agents[Editor Agents]
  manager-agent --> verifier-agent[Verifier Agent]
  verifier-agent --> human-input-tool[Human Input Tool]
  human-input-tool --> user
  manager-agent --> tool-panel[Tool Panel]
  editor-agents --> python-dsl-tool-calls[Python DSL Tool Calls]
  python-dsl-tool-calls --> backend-parser[Backend Parser]
  backend-parser --> reflection-recovery[Reflection Recovery]
  reflection-recovery --> manager-agent
  claude-sonnet[Claude Sonnet] --> replit-agent
  auxiliary-models[Auxiliary Models] --> replit-agent
  replit-agent --> langsmith-observability[LangSmith Observability]
  langsmith-observability --> trace-replay[Trace Replay]

Components

Data flow

A user prompts Replit Agent. The manager coordinates editor agents that modify code through exposed Replit tools, while the verifier checks application progress through screenshots, static checks, console errors, and app interaction. For tool calls, the model reasons in context and emits stripped-down Python DSL, the backend parses it, invalid calls are retried with error feedback, and periodic reflection asks whether the trajectory is making progress. Production traces record state at every agent step for debugging and replay.

Why this architecture

Replit's architecture tries to keep users engaged while preventing a catchall agent from drowning in tools. The team added multi-agent scope isolation after a ReAct-only loop hit too many tool errors, used code-generated DSL because LLMs are strong at code and function-calling APIs were too limited for complex tools, and invested early in trace observability because debugging production agents without replayable state is excruciating.

Models used: Claude 3.5 Sonnet · GPT-4 mini · fine-tuned embedding model

Scale

• Replit reported hundreds of thousands of agent runs in production.
  “tell you we had hundreds of thousands of runs in production as you know because are using your services as well so yes I think I think Alex on our infer team is very aware of that um you you talked a bunch about the”transcript, stated
• The team had about 90% of the required developer tools already inside Replit before exposing them as agent APIs.
  “for agents specifically we basically had maybe 90% of them already uh some of them were almost there we never thought of wrapping them as tools they were like somewhere hidden in the code base and they didn't have a clear API and we just”transcript, stated
• The DSL tool-invocation approach achieved about a 90% success rate for valid tool calls.
  “now but we are literally surely about 90% U success rate success rate of generating a valid tool call yeah especially because our tools are not easy to invoke like some of them of course you know we have one argument tools like everyone else uh but some”transcript, stated
• Replit tries to keep the multi-agent architecture to roughly three or four agents for debuggability.
  “so I think right now having like three or four of them makes it still relatively easy also to debug like right now in a lot of our traces we see that the manager is handing over the the editor and the editor is doing a series of actions and then the verifier kicks”transcript, stated
• Reflection checks run around every five defined agent steps.
  “steps and then we stop every at this point I think every five steps depending on Define steps in our agent and we should talk about trajectory land after I think it's it's going to be fun uh but like every few steps we stop the word and then we ask the agent to reason if”transcript, stated

Alternatives considered (roads not taken)

Full autonomy with the user out of the loop

Rejected because: Replit wanted users centered because long autonomous trajectories can crash or miss the goal.

Single catchall agent

Rejected because: A generic prompt with all tools works only up to a point and hits a ceiling for coding-agent complexity.

Provider function-calling APIs

Rejected because: They were elegant but limited and unreliable for Replit's complex tools.

Self-consistency over multiple LLM calls

Rejected because: It was too expensive and high-latency for their current production economics.

Patterns

multi-agent-orchestration (coding-agent)

Replit separates manager, editor, and verifier responsibilities so each sub-agent sees fewer tools and instructions.

human-in-the-loop-gate (product-control)

The verifier can ask the user when automated progress stalls, and the product keeps users engaged with questions and feedback.

tool-use-orchestration (developer-tools)

Replit exposes platform tools to agents and orchestrates them through a backend-parsed Python DSL.

structured-output-contract (tool-interface)

The backend validates generated DSL against schema compliance before accepting a tool invocation.

observability-tracing (agent-operations)

Replit records per-step state in LangSmith traces and replays traces to test bug fixes.

model-routing (model-portfolio)

Claude 3.5 Sonnet handles primary code generation/editing while smaller/other models handle compression, watchdog, and embedding tasks.

Tradeoffs

Keep the user engaged

Chose ask questions, show work, request feedback over pure full autonomy.

Rationale: User engagement helps recover from agent mistakes over long trajectories.

Generate tool calls as code

Chose Python DSL parsed by the backend over provider function-calling APIs.

Rationale: LLMs are strong at code generation, and schema-compliant DSL gave Replit more reliable tool calls.

Prioritize accuracy over latency

Chose higher-accuracy frontier model behavior over faster but less accurate responses.

Rationale: Replit judged wrong results as more frustrating than slower ones, while expecting model costs to improve over time.

Limit multi-agent complexity

Chose roughly three or four agents over larger swarms with more message passing.

Rationale: The current size remains debuggable; too many agents would make architecture behavior hard to understand.

Gotchas (what bit them)

Too many tools increased agent errors.

Root cause: A single broad tool scope creates too many opportunities for wrong choices.

Resolution: Replit introduced scoped sub-agents with minimal visible tools and instructions.

Function calling was not reliable enough for complex developer tools.

Root cause: Some Replit tools have many arguments and settings, making standard function calling brittle.

Resolution: The model emits Python DSL code that the backend parses and validates.

Production agent debugging is excruciating without trace replay.

Root cause: Long traces make it difficult to find where behavior diverged or failed.

Resolution: Replit stores full per-step state and replays trace states against patched codebases.

Coding-agent evals were insufficient for the whole user experience.

Root cause: Benchmarks helped with the core ReAct loop but not the end-to-end Replit Agent product.

Resolution: Replit combined partial evals, verifier work, internal use, early-access framing, and user feedback.

Evaluation

Replit combines partial coding-agent evals, verifier-agent checks, static checks, screenshots, app interaction, reflection, internal use, early-access feedback, and production trace replay. They explicitly note that existing evals help the core ReAct loop but do not yet cover the full product experience.

Metrics: valid tool-call success rate · agent runs in production · static check results · console errors · progress after every few steps · trace replay behavior · user feedback · core-loop eval pass rate

Outcomes

• Replit launched Replit Agent and reported hundreds of thousands of production runs.
• The product uses a scoped multi-agent architecture with manager, editor, and verifier roles.
• The Python DSL tool-call path achieved about 90% valid tool-call success.
• Claude 3.5 Sonnet became the primary code generation and editing model.
• LangSmith trace state and replay became central to production debugging.

The talk reports hundreds of thousands of production runs, about 90% valid tool-call success for generated DSL, about 90% of needed tools already existing inside Replit, and a preferred multi-agent size of roughly three or four agents for debuggability.