Online Reinforcement Learning for Code Completion at Scale

cursor · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production code completion

Cursor needed code-completion suggestions that improve developer flow at massive request volume by showing fewer noisy suggestions and increasing acceptance.

Constraints:

• Cursor Tab handles more than 400 million requests per day.
• Noisy suggestions disrupt developer flow.
• The system optimizes suggestion display probability against a target threshold.
• Online RL required rapid on-policy data collection and model deployment.

Architecture

Cursor implemented online reinforcement learning with policy gradients, on-policy production data collection, and rapid deployment cycles to optimize when Cursor Tab shows code-completion suggestions.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  cursor-tab["Cursor Tab"]
  production-feedback-stream["Production Feedback Stream"]
  policy-gradient-training["Policy Gradient Training"]
  rapid-deployment-pipeline["Rapid Deployment Pipeline"]
  cursor-tab --> production-feedback-stream
  production-feedback-stream --> policy-gradient-training
  policy-gradient-training --> rapid-deployment-pipeline
  rapid-deployment-pipeline --> cursor-tab

Components

Data flow

Cursor Tab serves suggestions, production acceptance behavior becomes on-policy feedback, training optimizes the policy gradient for whether to show suggestions above a target acceptance threshold, and rapidly deployed models update production behavior.

Why this architecture

Online reinforcement learning directly optimized the production behavior Cursor cared about: showing fewer low-value completions while increasing accepted suggestions.

Models used: Cursor Tab LLM · Policy gradient reinforcement learning

Scale

• Cursor Tab handles more than 400 million requests per day.
  “handling over 400 million requests per day.”summary, stated
• The online learning cycle takes 1.5-2 hours.
  “1.5-2 hour turnaround cycle.”summary, stated
• The resulting model reduced shown suggestions by 21%.
  “21% reduction in suggestions shown”summary, stated
• The resulting model increased accept rate by 28%.
  “increasing the accept rate by 28%”summary, stated

Alternatives considered (roads not taken)

Show more suggestions and rely on users to ignore bad ones

Rejected because: Noisy suggestions disrupted developer flow.

Offline-only model updates

Rejected because: The approach required on-policy data and rapid production deployment cycles.

Patterns

eval-driven-development (online-learning)

Production acceptance feedback directly drives model updates through online RL.

model-routing (serving-policy)

The model learns whether to show a completion based on predicted acceptance probability.

Tradeoffs

Show fewer suggestions

Chose Acceptance-thresholded display policy over Maximizing suggestion volume.

Rationale: Developer flow improved when noisy suggestions were reduced and accept rate rose.

Use online reinforcement learning

Chose Policy gradients from real-time user feedback over Static offline supervised tuning.

Rationale: The production objective was interaction-driven suggestion acceptance.

Gotchas (what bit them)

Completion quality is not only about generating more code.

Root cause: Too many low-value suggestions interrupt developer flow.

Resolution: Optimize the display policy to suppress suggestions below an acceptance threshold.

Online RL needs deployment infrastructure, not just an algorithm.

Root cause: On-policy data becomes stale unless models can be updated quickly.

Resolution: Build rapid deployment and on-policy data collection with 1.5-2 hour turnaround.

Evaluation

Cursor measured suggestions shown, accept rate, request volume, and model-update turnaround.

Metrics: Suggestions shown · Accept rate · Requests per day · RL turnaround cycle

Outcomes

• Suggestions shown dropped by 21%.
• Accept rate increased by 28%.
• The system handles more than 400 million requests per day.
• Online RL turnaround is 1.5-2 hours.

Cursor Tab runs at more than 400 million requests per day with 1.5-2 hour online learning cycles.