Building and Deploying Production LLM Code Review Agents: Architecture and Best Practices

ellipsis · Tech · 2024 · source Agentic: yes · promotional bias: med · summary-only (no fetched source) · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Automated software code review

Ellipsis needed production code-review agents that could find useful pull-request issues without flooding developers with false positives, while integrating with GitHub repositories and adapting to customer feedback.

Constraints:

• The fetched source is an unusable blog shell, so this extraction relies on the ZenML summary.
• Code-review comments must be accurate enough for developers to trust them.
• Repository search has to work across code structure, file-level architecture, and large contexts.
• Production review workflows need graceful failure modes when LLMs, vector stores, or subagents fail.

Architecture

Ellipsis routes GitHub events into asynchronous workflows, clones repositories, runs multiple specialized Comment Generator agents in parallel, filters their candidate comments through deduplication, confidence, hallucination, and editing stages, and reuses a Code Search subagent backed by keyword/vector retrieval, incremental indexing, language-server tools, and automated evaluation.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  github-app[GitHub App] --> workflow-queue[Workflow Queue]
  workflow-queue --> repository-clone[Repository Clone]
  workflow-queue --> review-agent[Review Agent]
  repository-clone --> incremental-indexer[Incremental Indexer]
  repository-clone --> review-agent[Review Agent]
  review-agent --> comment-generators[Comment Generators]
  review-agent --> code-search-subagent[Code Search Subagent]
  comment-generators --> filtering-pipeline[Filtering Pipeline]
  code-search-subagent --> incremental-indexer[Incremental Indexer]
  code-search-subagent --> language-server-tools[Language Server Tools]
  filtering-pipeline --> github-app[GitHub App]

Components

Data flow

A GitHub PR event enters the workflow queue, the repository is cloned, specialized comment generators inspect the change in parallel, Code Search retrieves supporting context, filters remove duplicates and hallucinations, and only surviving comments are posted or surfaced with reasoning.

Why this architecture

Ellipsis decomposed code review into smaller independently benchmarkable agents because one large prompt made the task too hard, while asynchronous review allowed deeper retrieval and filtering. Feedback search and modular evals favored operational iteration over slow per-customer fine-tuning.

Models used: Claude 3.5 Sonnet · GPT-4o · o1 · LLM-based binary classifiers

Scale

• Mini-benchmarks are built around about 30 examples for new agent development.
  “writing initial prompts and tools, sanity checking examples, finding correctness measures, building mini-benchmarks with around 30 examples (often using LLM augmentation for "fuzz testing"), measuring accuracy, diagnosing failure classes with LLM auditors, updating agents through prompt tweaks”summary, stated
• Most commits change only a small percentage of chunks, allowing index synchronization in seconds.
  “code stored), fetching existing IDs from the vector DB namespace, comparing new versus existing IDs, deleting obsolete IDs, and only embedding/upserting chunks with new IDs. In practice, since most commits affect only a small percentage of chunks, synchronization takes only a couple of seconds.”summary, stated

Alternatives considered (roads not taken)

A single large review agent prompt.

Rejected because: The team found it better to split difficult review work into simpler subagents that could be benchmarked independently.

Per-customer fine-tuned models for feedback.

Rejected because: Embedding search over historical thumbs up/down feedback updated behavior faster and was easier to maintain.

Re-embedding the whole repository on every commit.

Rejected because: Chunk hashes let the system delete obsolete vectors and embed only new chunks.

Patterns

multi-agent-orchestration (agent architecture)

Specialized comment generators run in parallel and are filtered into final code-review comments.

rag-retrieval (retrieval)

The Code Search subagent combines keyword and vector retrieval over chunked code and file summaries.

confidence-thresholding (quality control)

Candidate comments are thresholded by confidence before reaching developers.

eval-driven-development (evaluation)

New agents are developed through benchmarks, LLM auditors, failure-class diagnosis, and repeated prompt/tool updates.

fallback-degradation (reliability)

Failures are isolated at model, tool, and agent levels so the review can degrade instead of collapsing.

Tradeoffs

Review decomposition

Chose Dozens of small specialized agents over One large review prompt.

Rationale: Smaller agents are easier for the LLM to solve, benchmark, and optimize independently.

Feedback adaptation

Chose Embedding search over historical reactions over Per-customer fine-tuning.

Rationale: The approach reflects feedback immediately and avoids slower retraining loops.

Production robustness

Chose Layered graceful degradation over Assuming test performance predicts runtime behavior.

Rationale: Probabilistic LLM behavior requires retries, fallback models, tool isolation, and transparent user messaging.

Gotchas (what bit them)

False positives are the central trust failure in AI code review.

Root cause: Parallel generators can overlap, hallucinate, or produce weak comments without downstream filtering.

Resolution: Add deduplication, confidence, hallucination, and editing filters before posting comments.

Large code-search contexts can degrade LLM performance.

Root cause: Traditional retrieval can include code that ranks well but is not useful for the current agent trajectory.

Resolution: Use an LLM binary classifier after vector search to keep only useful retrieved pieces.

LLMs are bad at exact code coordinates.

Root cause: They often miss column numbers and line numbers for language-server calls.

Resolution: Let the agent specify symbol names and fuzzy-match to the closest symbol.

Evaluation

Ellipsis combines deterministic edit checks, LLM-as-judge comparisons to ground truth, mini-benchmarks, LLM-augmented fuzzing, trajectory auditing, and production edge-case sampling.

Metrics: Mini-benchmark accuracy · False positive reduction · Code Search judgment quality · Agent failure classes · Production edge cases

Outcomes

• Ellipsis built a multi-agent code-review system with a reusable Code Search subagent.
• The system filters raw comments with deduplication, confidence, hallucination, and editing stages.
• The case reports significant qualitative improvements but no numeric false-positive or acceptance-rate results.

The source provides operational design detail, mini-benchmark size, and seconds-level incremental indexing claims, but not customer-level volume or false-positive reduction metrics.