AI-Powered Code Review Platform Using Abstract Syntax Trees and LLM Context

baz · Tech · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI-assisted software development

Baz needed an AI reviewer that understood codebase context, not just isolated diffs, as coding agents generated more code requiring review.

Constraints:

Large repositories cannot be naively dumped into an LLM context
Agent tools must be security-reviewed rather than arbitrary
LLM quality is probabilistic and needs benchmark-driven release confidence

Architecture

Baz builds code-review value around context: AST traversal and repository knowledge compact relevant code into LLM context, a constrained agent uses reviewed integrations, and traces plus benchmarks guard probabilistic review quality.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  pull-request-intake["Pull request intake"]
  ast-code-graph["AST code graph"]
  repository-knowledge-base["Repository knowledge base"]
  context-packer["Context packer"]
  constrained-agent-container["Constrained agent container"]
  tool-integrations["Tool integrations"]
  review-generator["Review generator"]
  tracing-and-benchmarks["Tracing and benchmarks"]
  pull-request-intake --> ast-code-graph
  pull-request-intake --> repository-knowledge-base
  ast-code-graph --> context-packer
  repository-knowledge-base --> context-packer
  context-packer --> constrained-agent-container
  constrained-agent-container --> tool-integrations
  tool-integrations --> constrained-agent-container
  constrained-agent-container --> review-generator
  review-generator --> tracing-and-benchmarks
  tracing-and-benchmarks --> review-generator

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Pull request intake`	Receives diffs, tickets, CI context, and repository metadata for review.	`AST code graph`, `Repository knowledge base`
`AST code graph`	Uses AST traversal to retrieve definitions and dependencies relevant to the diff.	`Context packer`
`Repository knowledge base`	Stores product, codebase, and integration knowledge behind the platform.	`Context packer`
`Context packer`	Compacts only relevant code and metadata into LLM context.	`Constrained agent container`
`Constrained agent container`	Lets the LLM choose reviewed integrations such as tickets and CI logs without arbitrary HTTP access.	`Tool integrations`, `Review generator`
`Tool integrations`	Fetch ticketing, CI, and observability context through deterministic APIs.	`Constrained agent container`
`Review generator`	Produces code-review findings from compacted context.	`Tracing and benchmarks`
`Tracing and benchmarks`	Use OpenTelemetry traces and tagged datasets to evaluate changes before release.	`Review generator`

Data flow

The platform retrieves definitions and relevant context with AST/code-graph techniques, lets the LLM use deterministic integrations, and validates changes with benchmarks and observability traces.

3 supporting quotes

“using a I can bring the definitions and all the things that it uses but only those into the context suddenly the LLM sees everything much more clearly because it has only the things that it needs.”transcript, stated

“the NLM can decide, hm, this is a ticket. Which ticketing system are you using? It's a step. And then there's a deterministic API that says linear and then DLM says, okay, let's fetch the ticket from linear.”transcript, stated

“without running before you release every change on a big data set which you have tagged which is work that engineers hate and I hate it but it's very important um you won't have the confidence at releasing”transcript, stated

Why this architecture

The architecture limits tools and context because naive repository dumping overwhelms the model and integrations create prompt-injection surfaces.

“Now the uh problem here is that even the best developer I know if you just let him read so many Python files or so many you know it's not just Python a repository has YAML and configs and all sorts of things”transcript, stated

“we've had people just connect and um open a pull request and the different the pull request is write me a React component that does blah blah blah blah blah and it would expect the LM to just grab that”transcript, stated

Models used: Claude 3.5 Sonnet · GPT-4 · GPT-3.5

Scale

The source emphasizes benchmark-driven releases with accepted 80-90% targets rather than expecting deterministic 100% LLM behavior.
“the benchmark would not pass at 100%. Okay, this is not a test and LLM are statistical machines. So that means that um I mean we have benchmarks that aim for 80 to 90%.”transcript, implied
Baz accepts benchmark pass bands around 80-90% rather than treating LLM review as deterministic.
“the benchmark would not pass at 100%. Okay, this is not a test and LLM are statistical machines. So that means that um I mean we have benchmarks that aim for 80 to 90%.”transcript, stated
The platform uses OpenTelemetry-style traces for inspection across observability backends.
“we integrated uh we select the data it doesn't really matter we selected uh it's all based on open telemetry so if you use sentry or new or even uh the raw uh open telemetry stuff with uh Jagger and uh Grafana it should also work”transcript, stated

Alternatives considered (roads not taken)

Dump the whole repository or diff into the LLM

Rejected because: Baz uses AST-based context compaction because the model needs only relevant definitions and dependencies.

Differentiate on prompts or model access

Rejected because: The company argues defensible value comes from proprietary context rather than prompts or models.

“context is the only mode that you can really create in today's AI landscape because if it's a prompt, someone will hack you and and extract the prompt and if it's the model, a new model will come out”transcript, stated

Arbitrary agent HTTP calls to user-provided sites

Rejected because: Baz restricts tool integrations because connected PR content is a prompt-injection surface.

“So um it's crazy how many bad actors or not bad just mischievous people who want to try and and and uh beat the system. Um so we don't allow just like we don't allow making HTTP calls to whatever site you provide.”transcript, stated

No benchmark before LLM-review changes ship

Rejected because: The source says release confidence depends on running changes against a tagged benchmark dataset.

Patterns

`context-compaction` (context-engineering)

AST traversal narrows the LLM input to relevant definitions and references instead of sending whole repositories.

`tool-use-orchestration` (agent-architecture)

The agent decides when a ticket reference requires a supported Linear/Jira lookup, while deterministic APIs perform the actual fetch.

`observability-tracing` (operations)

Baz traces LLM inputs and outputs with OpenTelemetry-compatible tooling so hallucinations and quality failures can be debugged.

“we integrated uh we select the data it doesn't really matter we selected uh it's all based on open telemetry so if you use sentry or new or even uh the raw uh open telemetry stuff with uh Jagger and uh Grafana it should also work”transcript, stated

`eval-driven-development` (evaluation)

Every release is measured against tagged benchmark data, with expected success rates reflecting statistical model behavior rather than deterministic tests.

`safety-policy-enforcement` (security)

Baz rejects arbitrary user-provided tool calls because pull-request diffs can be used as prompt-injection attempts.

Tradeoffs

Where to build defensible value in an AI code-review product.

Chose context construction around the codebase over prompt secrecy or exposing model choice as the feature.

Rationale: The CTO argues prompts can be extracted and models will improve, while context remains hard to build.

How to treat benchmark pass rates for LLM features.

Chose statistical quality bars over expecting deterministic 100% test passes.

Rationale: The team explicitly accepts that LLM benchmarks may target 80-90% or 40-50% depending on task difficulty.

“the benchmark would not pass at 100%. Okay, this is not a test and LLM are statistical machines. So that means that um I mean we have benchmarks that aim for 80 to 90%.”transcript, stated

Gotchas (what bit them)

Naive context loading overwhelms the LLM and hides useful structure.

Root cause: Repositories are graphs of calls, definitions, configs, and dependencies, not ordered piles of files.

Resolution: Baz uses AST traversal to pull only related definitions and references for the changed code.

LLMs can hallucinate full code reviews when expected diff data is missing.

Root cause: An early parameter mismatch left the model with an empty code field but no hard guard against fabricating a plausible PR.

Resolution: The team added tracing, observability, and benchmarks before design-partner releases.

“since it has a field called code which was empty, the LM just decided to I know load from memory a pull request in Python which we didn't write back then. It was just Rust.”transcript, stated

Agent integrations become a prompt-injection surface.

Root cause: Users can open pull requests whose diffs try to instruct the LLM to perform unrelated actions.

Resolution: Baz limits the agent to supported, reviewed integrations and structured deterministic APIs.

Evaluation

Baz uses LLM tracing plus tagged benchmark datasets before releases, accepting non-deterministic success bands rather than binary test semantics.

Metrics: benchmark success bands · trace inspection · hallucination debugging

“once you have that you have to build a benchmark and what is a benchmark that's that's like a a heavy topic to talk about but um without running before you release every change on a big data set”transcript, stated

Outcomes

Baz reviews AI-generated and human pull requests using codebase context, ticket context, and CI context.
The product supports AI-code-generation workflows where Codex writes code, Baz reviews it, and Codex iterates.

The source is an interview rather than a metrics-heavy case study; it emphasizes product architecture and usage patterns more than customer-scale numbers.

“then codex implements it opens a pull request buzz reviews it and then codeex go for a fixed round and then we get better code without the CEO really learning at the conventions here.”transcript, stated