Building a Production AI Code Review Agent with High Engineer Acceptance

doordash · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production AI code review for large engineering organizations

DoorDash needed an AI code reviewer that engineers would trust enough to act on, while catching critical cross-boundary, deletion, and silent-behavior issues that human reviewers predictably skim.

Constraints:

• generic or noisy review comments cause engineers to scroll past or mute the agent
• specialist reviewers miss architectural and cross-boundary issues when they lack the big picture
• whole-change reviewers spread attention too thin if asked to inspect everything deeply
• repo-specific review doctrine lives in incidents, Slack decisions, historical reviews, and code conventions
• production review cost, latency, and comment quality must be bounded across thousands of weekly PRs

Architecture

DoorDash's third-generation review system uses a lead scout to scan the diff for suspicious leads, then two deep reviewers verify only the leads that deserve investigation. It routes each PR to per-domain review profiles mined from AGENTS.md, historical PR reviews, Slack decisions, and incidents, runs with full repository access in remote VMs, applies a disprove-it pass before posting, and can hand review findings to a fixer agent that patches the PR in an isolated worktree.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  pr-trigger[PR trigger] --> lead-scout[Lead scout]
  lead-scout --> deep-reviewers[Deep reviewers]
  deep-reviewers --> review-profiles[Review profiles]
  deep-reviewers --> remote-repository-vm[Remote repository VM]
  deep-reviewers --> disprove-it-pass[Disprove-it pass]
  disprove-it-pass --> reporter[Reporter]
  reporter --> github-pr-thread[GitHub PR thread]
  github-pr-thread --> fixer-agent[Fixer agent]
  fixer-agent --> remote-repository-vm
  eval-set[Eval set] --> review-profiles
  eval-set --> deep-reviewers

Components

Data flow

A PR opens and triggers a review. The lead scout reads the diff and emits investigation leads. Deep reviewers load only the relevant review profiles for the touched domain, inspect full-repository context in a remote VM, trace callers and sibling implementations, falsify candidate findings, and send grounded findings to a reporter. If a human or AI comment needs a fix, the fixer receives the original review context, changes the branch in a remote checkout, and pushes a normal PR diff.

Why this architecture

DoorDash optimized for trusted, precise comments rather than broad recall. Splitting noticing from verifying focuses attention on suspicious code, review profiles load only the doctrine that matters for a change, full-repo remote execution gives reviewers the context humans use, and the disprove-it pass keeps speculative comments out of the PR.

Models used: LLMs · code review agents · coding-agent harnesses · model-agnostic providers

Scale

• The agent reviews more than 10,000 pull requests per week across 56 onboarded repositories.
  “In a typical week, the agent reviews more than 10,000 pull requests across 56 onboarded repositories: Go backend services, iOS and Android apps, web frontends, infrastructure, data pipelines.”article, stated
• 60.2% of settled high and critical findings led engineers to change code before merge, across n=2,256 findings.
  “* 60.2% of settled high and critical findings result in the engineer changing code before merge (n=2,256), up from a 46% acceptance rate with our previous third-party review agent. We measure this on if the human acted on it or not.”article, stated
• The previous third-party agent had a 46% acceptance rate.
  “* 60.2% of settled high and critical findings result in the engineer changing code before merge (n=2,256), up from a 46% acceptance rate with our previous third-party review agent. We measure this on if the human acted on it or not.”article, stated
• Webhook-triggered findings were accepted at 59.0%.
  “* Webhook-triggered findings are accepted at 59.0%. That matters because these are not cherry-picked manual invocations. The agent runs automatically, on real PRs, and still produces comments engineers act on.”article, stated
• Average review cost was roughly $3, versus comparable products priced around $5 to $20 per review.
  “* A review costs roughly $3 on average today. Similar deep, agentic code review products with remote execution, repository access, and verification are publicly priced around $5 to $20 per review, depending on change size.”article, stated

  “* A review costs roughly $3 on average today. Similar deep, agentic code review products with remote execution, repository access, and verification are publicly priced around $5 to $20 per review, depending on change size.”article, stated
• Findings post about seven minutes after a PR opens on average.
  “* The agent posts findings about 7 minutes after a PR opens on average, usually before the first human reviewer has looked at the diff. Most engineers see the comment while the change is still fresh.”article, stated

Alternatives considered (roads not taken)

Fan out to narrow specialist reviewers

Rejected because: Specialists caught mechanical bugs but missed architectural and cross-boundary issues because none saw the full picture.

Two general-purpose reviewers inspect the whole change

Rejected because: Whole-change reviewers had too much to do and lost real findings because attention spread thin.

Optimize for catching everything

Rejected because: Surfacing every possible concern creates PR spam and causes engineers to ignore or mute the agent.

Dump all available context into the model

Rejected because: Full AGENTS.md, full diff, and README context produced confident-sounding but unfocused reviews.

Patterns

multi-agent-orchestration (review-pipeline)

DoorDash separates lead generation from verification, using a lead scout followed by two deep reviewers rather than a single monolithic reviewer.

retrieval-ranking (context-selection)

The agent routes PRs to only the domain review profiles relevant to the touched code, rather than loading all possible guidance.

eval-driven-development (quality-loop)

DoorDash uses real review misses and high-severity incidents as a small eval set for prompt, retrieval, model, and profile changes before production.

model-routing (cost-quality)

Because the workflow is staged and model-agnostic, DoorDash can use cheaper models for simple stages and stronger models for verification-heavy work.

durable-execution (bounded-runs)

The review runner uses per-agent soft and hard timeouts to bound stuck agent loops and return verified findings instead of wasting long runs.

Tradeoffs

Optimize for precision over recall

Chose Fewer, falsified, high-confidence comments over Surfacing every possible concern and letting humans filter.

Rationale: Trust and voluntary enablement mattered more than catching every possible issue, because a muted reviewer catches nothing.

Build internally

Chose DoorDash-owned review pipeline over A generic third-party reviewer.

Rationale: Owning the pipeline allowed DoorDash to encode internal doctrine, use full-repo context, swap models, and tune cost.

Stage review work by risk and difficulty

Chose Cheaper/simple steps plus stronger verification-heavy steps over One expensive model pass for every PR.

Rationale: Cost per successful review matters more than token price, and staged workflows make cost tunable without silently losing quality.

Use AI to close the fix loop but keep normal review

Chose Remote fixer commits as ordinary PR diffs over Removing engineer ownership after a finding.

Rationale: The fixer removes mechanical handoff while preserving CI and human review for the resulting change.

Gotchas (what bit them)

Specialized agents can miss architectural issues.

Root cause: Narrow reviewers lack cross-cutting context and do not see the bigger picture.

Resolution: Use a lead scout and deep reviewers that can inspect full change context and investigate suspicious leads.

More context can make reviews worse.

Root cause: Authoring docs contain setup, style, and general guidance that is noisy for review.

Resolution: Mine per-domain review profiles that keep only review-relevant invariants, rules, and historical lessons.

A stuck agent can burn tokens without making progress.

Root cause: Max-turn limits did not detect loops when the turn counter was not advancing.

Resolution: Add per-agent soft and hard timeouts so the system returns only already verified findings or kills the run.

Correct findings can still fail as review comments.

Root cause: Broad, weak, duplicate, or stale comments erode trust even when the underlying concern is real.

Resolution: Anchor comments to changed files and lines, give concrete behavior risk, collapse stale comments, and drop unactionable concerns.

Evaluation

DoorDash uses production acceptance rate as the lagging trust signal and a smaller incident/review-miss eval set for day-to-day development of prompts, retrieval, model choices, and review profiles.

Metrics: 60.2% settled high/critical finding acceptance · 59.0% webhook-triggered finding acceptance · review cost of roughly $3 · findings posted about seven minutes after PR open · small eval set from real review misses and high-severity incidents

Outcomes

• The current v3 architecture increased settled high/critical finding acceptance to 60.2%.
• The reviewer runs automatically on real PRs across 56 repositories and more than 10,000 PRs per week.
• Review findings typically arrive about seven minutes after PR open, often before a human reviewer has looked.
• The system can also fix issues from review comments through remote VM worktrees while preserving ordinary PR review.

DoorDash reports more than 10,000 weekly PR reviews across 56 repositories, 60.2% acceptance on high/critical findings, 59.0% webhook-triggered acceptance, roughly $3 per review, and about seven minutes to findings.