ML-Based Comment Ranker for LLM Code Review Quality Improvement

atlassian · Tech · year n/a · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM-powered code review quality control

Atlassian needed to reduce noisy, nit-picky, or factually wrong comments from an LLM code reviewer agent while preserving useful review coverage and adapting as foundation models, repositories, and user bases changed.

Constraints:

• LLM-generated code review comments could create negative user feedback when posted without filtering.
• Earlier heuristic filters improved precision but could not fully use Atlassian's proprietary interaction data.
• The ranker needed to generalize across repository rollouts, external beta customers, and different upstream LLMs.
• The quality signal had to be tied to developer behavior rather than subjective labels alone.
• The model had to be refreshed as usage data, code patterns, and foundation models changed.

Architecture

Atlassian inserted an ML comment ranker between the code reviewer agent's LLM-generated comments and the comments posted to pull requests. The code reviewer agent is triggered when a PR is created and uses the code diff, PR metadata, and Jira work item context to generate candidate comments. A fine-tuned ModernBERT binary classifier scores each raw textual comment for its likelihood of leading to code resolution. Comments must pass an A/B-optimized propensity threshold before posting. User actions after posted comments become ground truth labels, creating a refresh loop that expanded training data from about 10K to 53K comments and restored performance after broader repository rollout.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Trigger["Pull request trigger"] --> Agent["Code reviewer agent"]
  Agent --> LLM["LLM comment generator"]
  LLM --> Ranker["Comment ranker"]
  Ranker --> Gate["Threshold gate"]
  Gate --> Posted["Posted PR comments"]
  Posted --> Signal["Code resolution signal"]
  Signal --> Refresh["Training data refresh"]
  Refresh --> Ranker

Components

Data flow

When a PR is created, the code reviewer agent calls an LLM with the diff, PR title and description, and Jira context to generate original comments. Those comments are passed to the ModernBERT comment ranker, which predicts a 0-to-1 propensity score for code resolution. Comments above the threshold are posted to the PR. Later author code changes, captured as Bitbucket outdated status, become binary ground truth for training and refreshing the ranker.

Why this architecture

Atlassian separated comment generation from comment selection because prompt or heuristic filtering alone did not use the growing user-action data. A specialized ModernBERT classifier could learn from real code-resolution outcomes, act as a quality gate for different upstream LLMs, and be refreshed as product rollouts changed the data distribution.

Models used: ModernBERT · GPT-4o · Claude Sonnet 3.5 · Claude Sonnet 4 · Rovo Dev code reviewer agent

Scale

• The beta program had 10K+ monthly active users.
  “it already has 10K+ MAUs and helps review 43K+ PRs monthly for beta customers.”article, stated
• The code reviewer helped review more than 43K pull requests monthly for beta customers.
  “it already has 10K+ MAUs and helps review 43K+ PRs monthly for beta customers.”article, stated
• The training set expanded from about 10K to more than 53K review comments.
  “With more and diverse data signals collected from all Atlassian repos after that enablement (review comments in training data increased from ~10K to ~53K), we retrained the model”article, stated
• Fine-tuning the classifier can be completed within a few hours with GPU support.
  “Thanks to the GPU support from our ML platform team, fine-tuning those models can be finished within a few hours.”article, stated
• The code reviewer had more than 400 external active beta customers.
  “Code reviewer currently has over 400+ external active beta customers and their metrics are even better than metrics from internal users.”article, stated

Alternatives considered (roads not taken)

Post LLM-generated comments without filtering

Rejected because: Unfiltered LLM comments were noisy, nit-picky, or factually wrong and created negative user feedback.

Continue relying on heuristic and LLM-category filters

Rejected because: Heuristics helped precision but could not fully leverage proprietary data, modern ML, and transformer architectures.

Train only on limited repository data

Rejected because: When the product expanded to all eligible internal repositories, model precision dropped until retraining used broader data.

Patterns

confidence-thresholding (quality_control)

The ranker emits a 0-to-1 propensity score, and comments must exceed an A/B-optimized threshold before posting.

fine-tuning-adaptation (model_optimization)

Atlassian fine-tuned ModernBERT on proprietary comment-resolution data for the comment selection task.

eval-driven-development (evaluation)

The north-star metric is code resolution rate, linking model selection quality to whether developers actually change code after comments.

data-quality-loop (data)

User actions after comments become training labels, and broader repository rollout produced more diverse data for retraining.

fallback-degradation (quality_control)

The ranker provides a robust quality gate across upstream LLM switches, reducing dependence on a single generation model's behavior.

Tradeoffs

Optimize for code resolution rate

Chose Behavioral CRR as the primary success signal over Only subjective comment ratings or generic ML accuracy.

Rationale: CRR ties comment selection to whether developers made code changes after a comment, giving a scalable user-action signal.

Use raw comment text first

Chose LLM-generated comment content as the current ranker input over Immediately adding richer code, file, PR, and Jira features.

Rationale: The raw-text classifier performed well enough for production, while additional signals remain future work.

Add a specialized ML quality gate instead of relying only on the upstream LLM

Chose ModernBERT ranker as a post-generation filter over Prompting or swapping the generation LLM alone.

Rationale: The ranker complemented LLM generation and stayed robust across GPT-4o, Claude Sonnet 3.5, and Sonnet 4 transitions.

Gotchas (what bit them)

LLM review comments can harm UX when quality filtering is absent.

Root cause: Generated comments may be noisy, nit-picky, or factually wrong.

Resolution: Atlassian added an ML comment ranker to select useful comments before posting.

A model trained on limited internal data degraded when repository coverage expanded.

Root cause: The first model version used a limited set of repository signals that did not cover the broader rollout distribution.

Resolution: The team retrained on more diverse data from all eligible Atlassian repositories, increasing training comments from about 10K to 53K.

Resolution rate may miss valuable review comments that do not trigger immediate code changes.

Root cause: Code resolution is a binary behavioral outcome centered on code edits at the commented line.

Resolution: The source keeps future feature engineering open to add richer signals beyond raw comment text.

Evaluation

Atlassian evaluated the ranker using online code resolution rate, offline precision-like metrics, A/B threshold tests, hold-out datasets, before/after model-switch comparisons, internal human-review benchmarks, external beta metrics, and qualitative customer feedback.

Metrics: code resolution rate · threshold score optimized by A/B testing · hold-out dataset generalization · 40-45% CRR after ranker · human benchmark around 45% · CRR drop from ~40% to ~33% during broader rollout · PR cycle time reduction of 30%

Outcomes

• The code reviewer beta reached 10K+ monthly active users and reviewed 43K+ PRs monthly.
• PR cycle time dropped by 30% for pull requests resolved by the code reviewer agent compared with human-resolved PRs.
• The ranker increased CRR to 40-45%, close to the 45% human benchmark.
• Retraining after broader repository rollout recovered CRR from roughly 33% back to about 40%.
• CRR stayed consistent after the generation model changed from GPT-4o to Claude Sonnet 3.5.
• The ranker continued to complement newer LLMs, including Sonnet 4.

The source reports 10K+ MAUs, 43K+ PRs monthly, 400+ external active beta customers, 53K+ code-review comments in training data, and a 30% PR-cycle-time reduction.