Automated Code Reviews with LLMs

faire · E-commerce · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Automated code review

Faire wanted to automate generic pull-request review tasks so human reviewers could focus on architecture, product requirements, maintainability, and other high-context decisions.

Constraints:

some review checks need PR-specific context such as metadata, diff, build logs, and coverage reports
LLM output can be hallucinated, verbose, unrelated, or inconsistent
pull-request context changes over time, complicating review iteration and re-evaluation
duplicate or repeated automated reviews need to be avoided
human developers still need useful and timely comments rather than noisy review spam

Architecture

Faire built Fairey, an internal LLM orchestrator integrated with OpenAI Assistants APIs and GitHub webhooks. Fairey receives pull-request events, checks whether automated review criteria are met, starts Assistant chats, lets GPT call functions to fetch PR metadata, diffs, build logs, and coverage reports, checks whether the output is useful, and posts GitHub reviews with comments or suggested changes. Hidden metadata prevents duplicates, fixture snapshots preserve function-call outputs for later re-evaluation, and Gentrace/Comet/LangSmith-style eval tooling plus DX surveys measure review quality.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  github-pull-request["github-pull-request"] --> github-webhook-listener["github-webhook-listener"]
  github-webhook-listener["github-webhook-listener"] --> fairey-orchestrator["fairey-orchestrator"]
  fairey-orchestrator["fairey-orchestrator"] --> openai-assistant["openai-assistant"]
  openai-assistant["openai-assistant"] --> context-function-tools["context-function-tools"]
  context-function-tools["context-function-tools"] --> github-pull-request["github-pull-request"]
  context-function-tools["context-function-tools"] --> ci-coverage-build-data["ci-coverage-build-data"]
  context-function-tools["context-function-tools"] --> fairey-orchestrator["fairey-orchestrator"]
  fairey-orchestrator["fairey-orchestrator"] --> review-output-filter["review-output-filter"]
  review-output-filter["review-output-filter"] --> github-pull-request["github-pull-request"]
  fairey-orchestrator["fairey-orchestrator"] --> fixture-snapshot-store["fixture-snapshot-store"]
  fairey-orchestrator["fairey-orchestrator"] --> evaluation-tools["evaluation-tools"]
  fixture-snapshot-store["fixture-snapshot-store"] --> evaluation-tools["evaluation-tools"]
  developer-feedback["developer-feedback"] --> evaluation-tools["evaluation-tools"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`github-pull-request`	Source pull request with metadata, diff, status checks, and review thread surface.	`github-webhook-listener`, `fairey-orchestrator`
`github-webhook-listener`	Receives PR and check-run events and triggers eligible automated reviews.	`github-pull-request`, `fairey-orchestrator`
`fairey-orchestrator`	Internal LLM orchestration service that manages chat requests, function calls, RAG/context fetching, output checks, and review emission.	`github-pull-request`, `github-webhook-listener`, `openai-assistant`, `context-function-tools`, `review-output-filter`, `fixture-snapshot-store`, `evaluation-tools`
`openai-assistant`	Assistant configured with instructions and function tools for each automated review type.	`fairey-orchestrator`, `context-function-tools`
`context-function-tools`	Functions that fetch PR diffs, metadata, build logs, coverage reports, and other review-specific context.	`openai-assistant`, `fairey-orchestrator`, `github-pull-request`, `ci-coverage-build-data`
`ci-coverage-build-data`	CI check results, Jest/lcov coverage reports, build logs, and internal development portal data.	`context-function-tools`
`review-output-filter`	Checks whether Fairey has something useful to say before posting a review.	`fairey-orchestrator`, `github-pull-request`
`fixture-snapshot-store`	Stores function-call outputs as reusable snapshots so transient PR context can be re-evaluated later.	`fairey-orchestrator`, `evaluation-tools`
`evaluation-tools`	LLM evaluation tools and DX feedback used to score automated review quality quantitatively and qualitatively.	`fairey-orchestrator`, `fixture-snapshot-store`, `developer-feedback`
`developer-feedback`	Survey and pull-request author feedback that informs review quality and test-case filtering.	`evaluation-tools`

Data flow

GitHub webhooks notify Fairey about pull-request events and check-run outcomes. If a review’s criteria match, Fairey starts an OpenAI Assistant chat with the prompt template. GPT issues function calls such as fetching the diff, pull-request metadata, coverage reports, and build logs. Fairey executes those calls, submits results back to GPT, inspects the reply, and posts a GitHub review when it has useful comments or suggestions. Inputs and outputs are forwarded to evaluation tools, while function-call outputs can be stored as fixtures for repeatable future runs.

4 supporting quotes

“Faire has developed an LLM orchestrator service, which we’ve named Fairey”article, stated

“Fairey is also wired into Github’s webhooks, which deliver event payloads whenever interesting things happen on pull requests.”article, stated

“In order to produce a response, ChatGPT will issue function calls, which Fairey executes and submits back to ChatGPT”article, stated

“Then, we craft a Github Review using the reply, and post it to the pull request.”article, stated

Why this architecture

Fairey centralizes LLM orchestration so review logic does not sprawl across the codebase. The Assistants/function-call design lets GPT fetch only the PR-specific context it needs, rather than stuffing large diffs, logs, and reports into every prompt. GitHub webhooks make reviews event-driven, hidden metadata prevents duplicate comments, and fixture snapshots make transient PR data repeatable for prompt/model iteration.

4 supporting quotes

“creating a centralized service layer rather than embedding LLM calls throughout the codebase.”summary, stated

“This allows Fairey to decide when to fetch the data that’s needed, rather than having to send an overwhelming amount of extra information”article, stated

“To avoid duplicate reviews, each review left by Fairey will also include some hidden metadata”article, stated

“outputs of function calls are saved as snapshots for re-use in a later run.”article, stated

Models used: OpenAI Assistants API · ChatGPT · GPT · Jest · lcov · Gentrace · CometLLM · LangSmith · DX

Scale

Faire’s AI hackathon generated 92 submissions.
“The event was a raving success, with 92 projects submitted.”article, stated
The AI hackathon lasted three days.
“We ran a three-day AI Hackathon”article, stated
Frontend coverage checks run Jest with changedSince and coverage flags on pull requests.
“Our test coverage in pull requests is computed incrementally, meaning when we create a pull request, our CI system will execute the test suite with the [--changedSince](https://jestjs.io/docs/cli#--changedsince) flag, causing Jest to only execute tests that have a dependency on the source files in”article, stated
“We also set the [--coverage](https://jestjs.io/docs/cli#--coverageboolean) flag, which causes Jest to collect and output a coverage report, which can then be viewed in our internal development portal (see below for an example).”article, stated

Alternatives considered (roads not taken)

Expect human reviewers to handle all generic review tasks

Rejected because: Generic checks such as title quality, test coverage, style, and backward compatibility can be automated so humans focus on deeper review.

“there are also a range of more generic review requirements that can be considered without the extra context.”article, stated

Put all context in the main prompt

Rejected because: Function calls let Fairey fetch needed context on demand instead of sending overwhelming information up front.

“rather than having to send an overwhelming amount of extra information in the main prompt.”article, stated

Re-evaluate reviews against live PR state only

Rejected because: Pull-request data is transient, so fixtures preserve prior function outputs for repeatable evaluation.

“Pull requests reviews are quite transient in nature. Often, the information about the pull request relevant to the review will have changed by the time an engineer is ready to iterate on the behaviour and re-evaluate it.”article, stated

Early impact of automated reviews"

— article, stated

Patterns

`tool-use-orchestration` (agent-architecture)

GPT chooses function calls and Fairey executes them to retrieve PR-specific context.

“Each function call has a callback defined in the orchestration service”article, stated

`rag-retrieval` (retrieval)

The review pipeline retrieves proprietary PR data such as diff, metadata, logs, and coverage reports before producing comments.

“the information about the code changes being reviewed.”article, stated

`eval-driven-development` (evaluation)

Faire builds test cases from review inputs/outputs, reruns the review pipeline, and uses an LLM plus surveys to assess quality.

“build out a set of test cases, iterate on the reviews, and re-run them, computing a correctness score.”article, stated

`llm-as-judge` (evaluation)

An LLM evaluates the quality of automated code review outputs in the quantitative evaluation flow.

“Then, we use an LLM to assess the quality of the result.”article, stated

`human-in-the-loop-gate` (developer-feedback)

Pull-request authors provide qualitative feedback through DX surveys, and human reviewers retain high-context review work.

“qualitative assessment involves surveying the end user (our engineers).”article, stated

`agent-memory` (state)

Hidden metadata and thread history let Fairey avoid duplicate reviews and reuse previous function-call outputs.

“hidden metadata, so that we can check what’s already been covered.”article, stated

Tradeoffs

Automate generic review tasks

Chose LLM-powered comments for style, coverage, descriptions, build failures, and breaking changes over having humans spend review attention on every generic check.

Rationale: Automation frees engineering talent for complex, high-context review concerns.

“freeing up our talent to focus on the most impactful and complex parts of the review”article, stated

Fetch context through functions

Chose tool calls for targeted PR data over large static prompt context.

Rationale: Function calls avoid overloading the main prompt and let the assistant request only the data needed.

“decide when to fetch the data that’s needed”article, stated

Gotchas (what bit them)

LLM review output varies too much unless inputs are refined.

Root cause: LLMs can hallucinate, become verbose, or produce unrelated output when given unwieldy inputs.

Resolution: Use structured inputs, broad test cases, evaluation tooling, self-eval/CoT prompting, and survey feedback.

“GPT (generative pre-trained transformer) models are just predictive models, and they can make mistakes, outright hallucinate”article, stated

“unwieldy input gives unpredictable output.”article, stated

PR context can change before review behavior is re-evaluated.

Root cause: Pull requests are transient, so diffs, logs, and coverage reports may differ later.

Resolution: Save function-call outputs as fixture snapshots and replay them during later evaluation runs.

“the information about the pull request relevant to the review will have changed by the time an engineer is ready”article, stated

“extracting the results of the function calls into fixture files which we upload to storage.”article, stated

Evaluation

Faire evaluates automated reviews with quantitative LLM eval test cases and qualitative developer surveys. Inputs/outputs are forwarded to evaluation tools, test cases are rerun as prompts/models/data-fetching change, and DX solicits pull-request author feedback.

Metrics: correctness score · positive user satisfaction · high accuracy · DX survey responses · review output quality · coverage review usefulness

3 supporting quotes

“We assess the quality of the reviews using two signals: one quantitative, and the other qualitative.”article, stated

“The feedback, good or bad, is then plumbed into Gentrace”article, stated

“we’ve seen success — as measured by positive user satisfaction and high accuracy”article, stated

Outcomes

Faire created production automated review workflows from an AI hackathon project
automated reviews assess PR descriptions, style, test coverage, build failures, and backward-incompatible changes
Faire reports positive user satisfaction and high accuracy for automated reviews
human reviewers can spend more attention on architecture, requirements, maintainability, and code reuse

The source reports a three-day hackathon with 92 submissions and describes event-driven PR review automation over GitHub webhooks, CI coverage checks, and internal evaluation/test-case infrastructure.

“One of the projects where the Faire Engineering team ended up investing further was Automated Reviews.”article, stated

“Automation of reviews using LLMs helps achieve this by streamlining the review process, reducing review latency for simpler problems, and freeing up our talent to focus on the most impactful and complex parts of the review — such as correctly meeting product requirements, implementing a clean archit”article, stated