Building Production AI Coding Assistants and Agents at Scale

sourcegraph · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise coding assistants and agentic developer tooling

Sourcegraph needed to evolve from code search into production AI coding systems for large enterprise codebases. Cody addressed the chat-era problem of retrieving relevant code context for a single model response, while AMP addressed the agent-era problem of letting a coding agent gather context, call tools, iterate, and change software without being constrained by an application architecture built for chat.

Constraints:

• Large enterprise codebases require code navigation, branch-aware indexing, and build-system-specific precise code intelligence.
• Enterprise customers are privacy- and data-sensitive, limiting how proprietary code tokens can be reused for general model training.
• Earlier chat-style LLM applications needed external context engines because models could not reliably gather context themselves.
• Agentic coding changes the workflow from many people moving one SDLC loop to one developer driving many iterations in parallel.
• Traditional enterprise sales and compliance checklists can slow product discovery while the AI tooling landscape is still shifting.
• Code review and regression checks must catch subtle agent mistakes without recreating slow process-heavy workflows.

Architecture

Sourcegraph built on its code-search foundation in two generations. Cody combines code search and frontier LLMs as a chat-era assistant: a retrieval/context engine finds relevant code from large, messy codebases, then the model answers questions or generates code. AMP is a separate agent-era product rather than a Cody bolt-on: it wraps a frontier model in a loop with tools, lets the agent gather its own context, act on the world, and iterate, and treats agents/tools as composable functions inside application scaffolding. The product process mirrors the architecture: a small AMP team uses direct mainline ownership, regression-style examples instead of academic evals, and internal dogfooding to adapt quickly.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  enterprise-codebases["Enterprise codebases"] --> code-search-index["Code search index"]
  code-search-index --> precise-code-intelligence["Precise code intelligence"]
  code-search-index --> cody-context-engine["Cody context engine"]
  precise-code-intelligence --> cody-context-engine
  cody-context-engine --> chat-llm-assistant["Chat LLM assistant"]
  chat-llm-assistant --> enterprise-developers["Enterprise developers"]
  enterprise-developers --> amp-coding-agent["AMP coding agent"]
  amp-coding-agent --> primitive-tool-set["Primitive tool set"]
  amp-coding-agent --> frontier-model["Frontier model"]
  primitive-tool-set --> enterprise-codebases
  amp-coding-agent --> regression-workflow-examples["Regression workflow examples"]
  core-amp-team["Core AMP team"] --> amp-coding-agent
  core-amp-team --> regression-workflow-examples
  regression-workflow-examples --> amp-coding-agent
  amp-coding-agent --> enterprise-developers

Components

Data flow

The code-search layer indexes large codebases, branches, compiler-derived symbols, and language-agnostic navigation primitives. Cody uses that retrieval capability to place relevant enterprise code context into a chat LLM's context window for code generation or technical Q&A. AMP starts from user intent, enters a model-and-tool loop, gathers context with primitive tools, acts on code, communicates progress, and iterates. The team validates important workflows with regression examples, dogfoods the system, and treats application scaffolds as a way to generate new data for later model improvement.

Why this architecture

Sourcegraph split Cody and AMP because the model capability boundary moved. Cody needed a smart retrieval/context engine for chat models that lacked agency. AMP could be much simpler at the retrieval layer because robust tool use moved more of the search and planning intelligence into the model-and-tool loop. Keeping AMP as a ground-up product also let the team avoid enterprise checklist drag and explore a changed SDLC instead of fitting agents into old code-review and sales-process slots.

Models used: Claude 3 Sonnet · frontier LLMs · chat LLMs · GPT-5 · Kimi K2 · Gemini

Scale

• Sourcegraph serves on the order of hundreds of large enterprise and government customers.
  “we probably in the   order of like hundreds. So, like as a company,  we're primarily geared towards serving like larger   enterprise customers.”transcript, stated
• Customers include Fortune 500s, government agencies, growth-stage startups, public companies, banks, and global government entities.
  “like the, you know, Fortune 500s and government   agencies. Um, everyone ranging from like growth  stage startups, uh, like Stripe or OpenAI”transcript, stated
• Cody customer-reported productivity gains range from about 30% to 60%.
  “Cody's efficiency gains um they're  they range between like 30 and I think like 60%   uh efficiency gains in in development  productivity.”transcript, stated
• AMP was about five months old at the time of the talk, showing the compressed iteration cycle for the agent product.
  “Like it's crazy to think that AMP is only   five months old. It feels like five years  old.”transcript, stated
• The AMP core team was around 10 people.
  “it's like a really small uh core team uh probably  around like 10 people right now.”transcript, stated
• The speaker estimates about 80% of his development time is effectively reviewing agent output inside the editor.
  “I spend probably  like 80% of my development time like effectively   doing code review, right? Like reading through  agent output”transcript, stated

Alternatives considered (roads not taken)

Build AMP inside Cody

Rejected because: The team concluded that agentic LLM applications were a distinct architecture from chat-era assistants, so bolting agents into Cody would inherit the wrong product and context assumptions.

Use a sophisticated context/RAG engine as the primary intelligence layer for agents

Rejected because: For agents, much of the needed search/planning intelligence moved into the model, and primitive tools plus multiple hops could compete with more elaborate vector-search systems.

Run AMP through standard enterprise SaaS sales, feedback, and compliance processes immediately

Rejected because: The AI tooling landscape was changing too quickly for enterprise customers to know what they wanted, and premature checklist work would slow discovery.

Evaluate the chat-to-agent transition primarily through academic benchmark deltas

Rejected because: The capability shift was obvious in use, so the team used product dogfooding and workflow regression examples rather than treating it as an academic eval comparison.

Patterns

rag-retrieval (retrieval)

Cody combines Sourcegraph's code-search DNA with frontier LLMs so enterprise code context is retrieved and placed into the model context for code generation and technical Q&A.

tool-use-orchestration (agent-architecture)

AMP is framed as a loop around a model and tools, where the agent can gather context or act on the world instead of waiting for an external retrieval engine to provide all context.

eval-driven-development (quality)

Sourcegraph treats important workflows as regression examples, running them when making major AMP changes rather than relying on academic benchmark-style evals.

model-routing (model-architecture)

AMP is multi-model by design, but Sourcegraph treats agents and tools as composable functions rather than simply exposing a model selector.

data-quality-loop (training-feedback)

The speaker argues that application scaffolds create new input-to-output traces that can become training data for the next generation of models.

Tradeoffs

Start AMP as a new product rather than extend Cody

Chose A ground-up agent architecture over Bolting agent behavior into the existing chat-era Cody application.

Rationale: The team judged that robust tool use changed the application architecture enough that Cody's retrieval-heavy chat paradigm would constrain AMP.

Keep AMP small and exploratory

Chose A small research-lab-like core team over Prematurely scaling through enterprise compliance and sales checklists.

Rationale: Sourcegraph wanted to keep discovery speed while the agentic developer-tool landscape was unsettled.

Use practical regression workflows instead of academic evals

Chose Workflow examples run on major changes over Academic-style benchmark comparisons as the primary development guide.

Rationale: The chat-to-agent shift was large enough that product use made the capability jump obvious, while regression examples caught important workflow breakage.

Make authors own agent-assisted changes directly

Chose Push-to-main ownership and in-editor agent-output review over Default GitHub PR review as the main quality checkpoint.

Rationale: When one developer can drive many AI-assisted iterations, accountability and repeated close review of agent output may be more effective than process-heavy diffusion of responsibility.

Gotchas (what bit them)

A chat-era assistant architecture can become the wrong foundation once models can reliably use tools.

Root cause: Cody's architecture assumed context had to be gathered by an external context engine, while AMP's agents could gather context and act through tools.

Resolution: Sourcegraph restarted AMP from the ground up instead of bolting it into Cody.

Standard enterprise go-to-market can suppress awareness and slow learning in a rapidly changing AI market.

Root cause: Sourcegraph pushed Cody through enterprise B2B channels and listened to customers whose needs could shift within months.

Resolution: AMP is run more like a small research/product lab focused on discovering and inventing the future workflow.

Enterprise code data is valuable but cannot automatically become general training data.

Root cause: Customers are privacy- and data-sensitive, so proprietary code tokens cannot be rolled into models used outside the enterprise.

Resolution: Use the data to improve deployment inside enterprise contexts rather than assuming it can train a general model.

Agentic development can turn traditional code review into either a bottleneck or theater.

Root cause: Large PRs often get shallow LGTM review, while AI coding shifts much of the real review into the developer's interaction with agent output.

Resolution: The AMP team makes authors own changes, pushes directly to main, and reviews agent output repeatedly inside the editor.

Evaluation

Sourcegraph reports customer ROI/productivity metrics for Cody, uses workflow examples as regression tests for AMP, and relies heavily on internal dogfooding rather than academic eval benchmarks.

Metrics: Customer-reported Cody development-productivity gains of roughly 30-60%. · Workflow examples run when major AMP changes are made. · Internal user judgment that the agentic transition was obvious within minutes of use. · Developer time spent reviewing agent output inside the editor.

Outcomes

• Sourcegraph serves hundreds of enterprise/government customers with its code search and Cody products.
• Cody is deployed inside large enterprises and is associated with reported 30-60% productivity gains.
• AMP emerged as a separate agentic coding product around a new application architecture.
• The AMP team changed its product/process model toward small-team dogfooding, direct ownership, and regression workflow examples.

The case combines enterprise deployment scale for Sourcegraph/Cody with an intentionally small AMP core team exploring a newer coding-agent workflow.