Improving AI Code Review Bot Comment Quality Through Vector Embeddings

greptile · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI code-review comment quality and noise reduction

Greptile's AI code-review bot produced too many technically true but low-value comments, causing developers to ignore the bot. The team needed to reduce noisy nit comments while preserving critical findings, define an objective quality metric, and adapt quality thresholds to each team's subjective standards without tying the product to one fine-tuned model.

Constraints:

• LLM-generated comments are paid by the token and can become noisy
• developers ignored the bot when a pull request received too many comments
• quality was subjective and varied from team to team
• prompt engineering and few-shot examples reduced valuable comments or worsened behavior
• LLM-as-judge severity scoring added latency and was nearly random
• fine-tuning was rejected for cost, speed, and model-portability reasons

Architecture

Greptile built a feedback-driven semantic filter around its code-review bot. The bot still generates review comments, but developer behavior supplies the quality signal: whether comments are addressed in subsequent commits, plus upvotes and downvotes. Greptile embeds past comments per team and stores them in a vector database labeled by whether they were addressed/upvoted or downvoted. For each new generated comment, the system embeds the comment, compares it with that team's historical comment vectors, blocks it when it is similar to at least three unique downvoted comments, passes it when similar to at least three upvoted comments, and otherwise lets it through. This learns each team's bar for 'nittiness' without changing the base LLM.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Pull_request["Pull request"] --> AI_code_review_bot["AI code review bot"]
  AI_code_review_bot --> Candidate_comment_generator["Candidate comment generator"]
  Candidate_comment_generator --> Comment_embedding_generator["Comment embedding generator"]
  Comment_embedding_generator --> Similarity_filter["Similarity filter"]
  Similarity_filter --> AI_code_review_bot
  AI_code_review_bot --> Posted_comments["Posted comments"]
  Posted_comments --> Developer["Developer"]
  Developer --> Address_rate_evaluator["Address-rate evaluator"]
  Pull_request --> Address_rate_evaluator
  Developer --> Reaction_signal_collector["Reaction signal collector"]
  Address_rate_evaluator --> Team_feedback_store["Team feedback store"]
  Reaction_signal_collector --> Team_feedback_store
  Team_feedback_store --> Comment_embedding_generator
  Comment_embedding_generator --> Team_vector_database["Team vector database"]
  Team_vector_database --> Similarity_filter

Components

Data flow

A pull request arrives, the AI code-review tool generates candidate comments using codebase context, and Greptile evaluates historical quality signals by scanning later diffs for addressed comments and collecting reactions. Past comments are embedded and stored per team in a vector database. A new candidate comment is embedded, compared against the team's downvoted and upvoted clusters by cosine similarity, and filtered according to the three-similar-comments rule before comments are posted to the pull request.

Why this architecture

The final architecture uses observed developer behavior rather than trying to encode comment quality in a prompt or a universal judge. Because nits are subjective by team, Greptile stores feedback per team and filters by semantic similarity to prior team-specific reactions. This keeps the system model-agnostic, avoids another LLM inference call in the workflow, and lets the filter learn recurring nit clusters organically.

Models used: LLM-based code review generator · embedding model for comments · LLM-as-judge severity scorer (failed experiment)

Scale

• The source was adapted from a December 16, 2024 talk.
  “This post is adapted from a talk I gave at the Sourcegraph Dev Tools meetup at the Cloudflare office in San Francisco on December 16th, 2024.”article, stated
• A pull request with 20 changes could receive as many as 10 bot comments.
  “In a PR with 20 changes, it would leave as many as 10 comments, at which point the PR author would simply start ignoring all of them.”article, stated
• Initial analysis found about 19% good comments, 2% incorrect comments, and 79% nits.
  “We analyzed existing Greptile comments and found that ~19% were good, 2% were flat-out incorrect, and 79% were nits”article, stated
• The filter requires at least three unique downvoted or upvoted similar comments to decide.
  “If the comment had a cosine similarity exceeding some threshold with at least 3 unique downvoted comments, it would get blocked.”article, stated
• Within two weeks, address rate improved from 19% to over 55%.
  “Within two weeks of rolling out this feature, existing users saw address rate (percentage of Greptile’s comments that devs address before merging) go from 19% to 55+%.”article, stated

Alternatives considered (roads not taken)

Use GitHub thumbs up/down as the main quality signal

Rejected because: The team chose addressed-in-code behavior instead because it produced a more objective metric tied to developer action.

"There were two ideas here:

1. GitHub lets developers react to comments with 👍/👎. We could use this as the quality indicator."

Prompt engineer the base LLM to write fewer nits

Rejected because: Prompting reduced critical comments along with nits.

Few-shot examples of good and bad comments

Rejected because: The model learned superficial characteristics and made the bot worse.

LLM-as-a-judge severity score

Rejected because: The judge was nearly random and added another inference call, making the workflow slow.

Fine-tune the model

Rejected because: Fine-tuning was too costly, slow, and incompatible with Greptile's model-agnostic posture.

Patterns

agent-memory (memory)

Greptile stores per-team semantic histories of accepted and rejected comments so future filtering reflects that team's standards.

confidence-thresholding (quality)

A new comment is blocked or passed only when cosine similarity exceeds a threshold against at least three unique historical comments of one class.

data-quality-loop (quality)

Developer reactions and addressed-comment behavior continuously add examples to the team-specific filtering memory.

eval-driven-development (quality)

The team defined address rate as the north-star metric and iterated through failed and successful approaches against that metric.

llm-as-judge (quality)

Greptile tried an LLM judge for severity scoring, but the pattern failed in this subjective comment-quality setting.

Tradeoffs

Quality metric

Chose address rate from subsequent diffs over GitHub thumbs up/down alone.

Rationale: Address rate ties comment value to whether developers actually changed code before merging.

Post-generation semantic filter

Chose embedding-based comment filtering over changing the LLM with prompt engineering or fine-tuning.

Rationale: The filter reduced noise while preserving model-agnosticism and avoided prompt/fine-tune failure modes.

Conservative pass-through logic

Chose pass comments when neither or both similarity conditions are met over blocking uncertain comments.

Rationale: The rule blocks only comments with clear similarity to repeated downvoted clusters, avoiding aggressive suppression of potentially useful findings.

Team-level personalization

Chose per-team vector stores over one global nit filter.

Rationale: Nittiness is subjective, so filtering standards need to adapt to each team's expectations.

Accept imperfect but material improvement

Chose ship the embedding filter at 55+% address rate over waiting for a perfect comment-quality solution.

Rationale: The approach nearly tripled the address rate and was the most impactful noise-reduction technique despite remaining imperfect.

Gotchas (what bit them)

Prompting reduced critical comments along with nits.

Root cause: The model could not cleanly distinguish low-value comments from important comments through prompt instructions alone.

Resolution: Move comment suppression out of the prompt and into a feedback-driven semantic filter.

Few-shot examples made the bot worse.

Root cause: The LLM inferred superficial properties from examples rather than the deeper standard for useful comments.

Resolution: Use behavior-derived embeddings instead of relying on in-context examples.

LLM-as-judge was a bad fit for subjective comment severity.

Root cause: Severity judgments over comment-plus-diff pairs were subjective and context-dependent.

Resolution: Measure actual addressed comments and semantic similarity to team feedback instead of asking another LLM to score severity.

Quality was team-specific, not universal.

Root cause: Definitions of nits vary across teams and cannot be captured by a single global standard.

Resolution: Partition the vector feedback memory by team.

Fine-tuning would have reduced model portability.

Root cause: A fine-tuned solution would bind Greptile to a model family while adding cost and slower iteration.

Resolution: Use an external embedding/vector filter that remains model-agnostic.

Evaluation

Greptile evaluated code-review comment quality by tracking whether developers addressed comments in later commits before merge. This address rate replaced more subjective reactions as the primary performance metric, and the team used it to compare prompt engineering, few-shot examples, LLM-as-judge filtering, fine-tuning considerations, and the final embedding-based filter.

Metrics: address rate: percentage of generated comments developers address before merging · initial good comments: approximately 19% · initial incorrect comments: 2% · initial nit comments: 79% · LLM-as-judge severity threshold: below 7 filtered in failed experiment · embedding filter: at least 3 unique similar downvoted comments to block · address rate after rollout: 55+% within two weeks

Outcomes

• Greptile improved comment address rate from 19% to over 55% within two weeks of rollout.
• The final approach nearly tripled the address rate while remaining imperfect.
• Prompt engineering, few-shot examples, LLM-as-judge, and fine-tuning were rejected or failed for this problem.
• The system learned team-specific nit standards through vector embeddings of addressed, upvoted, and downvoted comments.
• The source frames comment noise reduction as an ongoing problem rather than solved completely.

This is a strong production lesson in letting user behavior supervise an LLM filter. It is especially useful for pattern synthesis because it shows a failed LLM-as-judge path and a successful semantic feedback memory path on the same problem.