AI-Orchestrated Code Review System at Scale

cloudflare · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Engineering code review and CI/CD quality gates

Cloudflare needed to reduce code-review bottlenecks across thousands of repositories without flooding engineers with noisy or unsafe AI comments.

Constraints:

Architecture

Cloudflare built a CI-native code-review orchestrator around OpenCode: a coordinator classifies merge-request risk, launches specialized reviewers, collects structured XML findings and JSONL logs, then applies approval rules with human break-glass escape.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  ci-review-trigger["CI review trigger"]
  opencode-coordinator["OpenCode coordinator"]
  risk-tier-classifier["Risk-tier classifier"]
  reviewer-sessions["Reviewer sessions"]
  structured-findings["Structured findings"]
  jsonl-event-logs["JSONL event logs"]
  coordinator-decision["Coordinator decision"]
  human-break-glass-override["Human break-glass override"]
  ci-review-trigger --> opencode-coordinator
  opencode-coordinator --> risk-tier-classifier
  risk-tier-classifier --> reviewer-sessions
  opencode-coordinator --> reviewer-sessions
  reviewer-sessions --> structured-findings
  structured-findings --> coordinator-decision
  jsonl-event-logs --> coordinator-decision
  coordinator-decision --> human-break-glass-override
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
CI review trigger Starts AI review from the merge-request pipeline. OpenCode coordinator
OpenCode coordinator Classifies risk, spawns reviewer sessions, gathers findings, and decides approval posture. Risk-tier classifier, Reviewer sessions, Structured findings
Risk-tier classifier Chooses review depth so small changes do not burn maximum model capacity. OpenCode coordinator, Reviewer sessions
Reviewer sessions Run specialized security, performance, code quality, documentation, release, and compliance reviews. Structured findings
Structured findings Returns XML findings from reviewers without constraining their internal tools. Coordinator decision
JSONL event logs Persist crash-tolerant debug and review events. Coordinator decision
Coordinator decision Produces approve, approve-with-comments, or block decisions with a bias toward approval. Human break-glass override
Human break-glass override Lets human reviewers force approval when needed. Coordinator decision

Data flow

Merge requests enter a coordinator that classifies risk, launches up to seven specialized reviewers, accepts structured XML findings, and produces a review decision.

“we launch up to seven specialised reviewers covering security, performance, code quality, documentation, release management, and compliance with our internal Engineering Codex.”article, stated
“The coordinator doesn't see or control what tools the sub-reviewers use. They are free to read source files, run grep, or search the codebase as they see fit, and they simply return their findings as structured XML”article, stated

Why this architecture

The orchestration, filtering, and human escape hatch exist because naive LLM review flooded engineers with vague or hallucinated comments.

“The results were exactly as noisy as you might expect, with a flood of vague suggestions, hallucinated syntax errors, and helpful advice to "consider adding error handling" on functions that already had it.”article, stated
“If a human reviewer comments break glass, the system forces an approval regardless of what the AI found.”article, stated

Models used: OpenCode-backed coding agents · coordinator agent · specialized reviewer agents

Scale

Alternatives considered (roads not taken)

Always run every specialized reviewer

Rejected because: Cloudflare uses risk tiers because a one-line typo fix does not justify seven expensive agents.

“You don't need seven concurrent AI agents burning Opus-tier tokens to review a one-line typo fix in a README. The system classifies every MR into one of three risk tiers”article, stated
“Security-sensitive files: anything touching auth/, crypto/, or file paths that sound even remotely security-related always trigger a full review”article, stated

Strictly block on uncertain findings

Rejected because: The coordinator biases toward approval with comments rather than blocking when evidence is weak.

“The bias is explicitly toward approval, meaning a single warning in an otherwise clean MR still gets approved_with_comments rather than a block.”article, stated

Passing massive merge-request prompts as command-line arguments

Rejected because: Cloudflare passed prompts through stdin to avoid process argument limits on large merge requests.

“We pass the coordinator prompt via stdin rather than as a command-line argument, because if you have ever tried to pass a massive merge request description full of logs as a command-line argument”article, stated

Ordinary JSON blobs for crash-time logs

Rejected because: Cloudflare used JSONL event logging because early exits can lose ordinary JSON debug logs.

“This is especially problematic if your application exits early before it has a chance to close everything out and write a valid JSON blob to disk — and this is often when you need the debug logs most.”article, stated

Patterns

multi-agent-orchestration (orchestration)

A coordinator launches specialized review agents in parallel and consolidates their findings into one structured review.

“we launch up to seven specialised reviewers covering security, performance, code quality, documentation, release management, and compliance with our internal Engineering Codex.”article, stated

structured-output-contract (reliability)

Sub-reviewers return structured XML findings, while the coordinator deduplicates, filters, assigns severity, and posts a single review comment.

“The coordinator doesn't see or control what tools the sub-reviewers use. They are free to read source files, run grep, or search the codebase as they see fit, and they simply return their findings as structured XML”article, stated

cost-aware-routing (operations)

The system classifies merge requests by risk tier so trivial changes do not consume the same agent budget as security-sensitive or large changes.

“You don't need seven concurrent AI agents burning Opus-tier tokens to review a one-line typo fix in a README. The system classifies every MR into one of three risk tiers”article, stated

human-in-the-loop-gate (safety)

A human reviewer can force approval with a break-glass comment for hotfixes or AI/provider failures, and the override is tracked in telemetry.

“If a human reviewer comments break glass, the system forces an approval regardless of what the AI found.”article, stated

Tradeoffs

How many reviewers to run per merge request.

Chose risk-tiered reviewer depth over always running the full specialist set.

Rationale: Security-sensitive or large diffs get full review, while small low-risk changes avoid unnecessary latency and token cost.

“Security-sensitive files: anything touching auth/, crypto/, or file paths that sound even remotely security-related always trigger a full review”article, stated

How aggressive the bot should be when findings are uncertain.

Chose bias toward approval and signal over noise over blocking every warning or producing many speculative comments.

Rationale: The system sits in the shipping path, so it keeps findings low-volume and blocks only serious problems or safety risks.

“The bias is explicitly toward approval, meaning a single warning in an otherwise clean MR still gets approved_with_comments rather than a block.”article, stated

How to preserve debugability during early process exits

Chose JSONL event logging over one large JSON blob written at process shutdown.

Rationale: Append-friendly JSONL survives the failures that are most likely to need debugging.

“This is especially problematic if your application exits early before it has a chance to close everything out and write a valid JSON blob to disk — and this is often when you need the debug logs most.”article, stated

Gotchas (what bit them)

Naive LLM code review produced noisy and hallucinated comments.

Root cause: Feeding raw diffs into a generic model lacked repository context, task specialization, and filtering.

Resolution: Use specialized reviewers plus a coordinator that deduplicates, judges severity, and filters speculative findings.

“The results were exactly as noisy as you might expect, with a flood of vague suggestions, hallucinated syntax errors, and helpful advice to "consider adding error handling" on functions that already had it.”article, stated

Large merge-request prompts hit process argument limits.

Root cause: Passing a massive merge-request description as a command-line argument can trigger Linux ARG_MAX limits.

Resolution: Spawn OpenCode as a child process and pass the coordinator prompt through stdin.

“We pass the coordinator prompt via stdin rather than as a command-line argument, because if you have ever tried to pass a massive merge request description full of logs as a command-line argument”article, stated

Crash-time debug logs are easiest to lose with ordinary JSON blobs.

Root cause: A process that exits early may never close and write a valid JSON array.

Resolution: Use JSONL so every output line is independently parseable and useful for partial failure debugging.

“This is especially problematic if your application exits early before it has a chance to close everything out and write a valid JSON blob to disk — and this is often when you need the debug logs most.”article, stated

Evaluation

Cloudflare reports production telemetry over the first month across review runs, merge requests, repositories, duration, cost, overrides, and findings.

Metrics: 131,246 review runs · 48,095 merge requests · 5,169 repositories · median review time 3 minutes 39 seconds · 288 break-glass overrides, 0.6% of merge requests · average review cost $1.19 · 159,103 findings

“the system completed 131,246 review runs across 48,095 merge requests in 5,169 repositories.”article, stated

Outcomes

Cloudflare engineering blog; production telemetry is detailed but self-reported.

“The system produced 159,103 total findings across all reviews, broken down as follows:”article, stated