Building Production-Ready LLMs for Automated Code Repair: A Scalable IDE Integration Case Study

replit · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production code-repair model training and IDE integration

Replit needed a production-ready model that could turn Language Server Protocol diagnostics into reliable code repairs inside its IDE, where deterministic LSP fixes covered only a small share of Python diagnostics and general code models were not trained on Replit's session events.

Constraints:

• The model had to operate on real IDE events rather than only natural-language coding prompts.
• LSP diagnostics were abundant, but only a minority had deterministic CodeActions.
• Training data had to be reconstructed from edit streams, events, and project snapshots at diagnostic time.
• Generated repairs needed to be parseable, applicable, and cheap enough for IDE inference.
• Evaluation had to cover both public programming benchmarks and Replit's own real-world repair distribution.
• Production latency and cost pushed the team toward a smaller specialized model rather than only frontier API models.

Architecture

Replit built Code Repair 7B as an end-to-end LLMOps pipeline around IDE-native data. Operational transformations, session events, LSP diagnostics, and project snapshots reconstruct the filesystem at the moment of an error; filtering removes deterministic or unsuitable diagnostics; a serverless verification step reruns Ruff/Pyright diagnostics; large code LLMs plus DSPy synthesize numbered line diffs; regex/application/LLM filters keep usable examples; DeepSeek-Coder-Instruct-v1.5 is fine-tuned on MosaicML infrastructure; and the model is evaluated against both a LeetCode-derived benchmark and a held-out Replit repair benchmark.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  replit-ide-session[Replit IDE session] --> operational-transformation-stream[Operational transformation stream]
  replit-ide-session --> session-event-timeline[Session event timeline]
  operational-transformation-stream --> filesystem-reconstruction-pipeline[Filesystem reconstruction pipeline]
  session-event-timeline --> filesystem-reconstruction-pipeline
  lsp-diagnostics-dataset[LSP diagnostics dataset] --> filtering-pipeline[Filtering pipeline]
  filtering-pipeline --> training-example-corpus[Training example corpus]
  filesystem-reconstruction-pipeline --> serverless-diagnostic-verifier[Serverless diagnostic verifier]
  serverless-diagnostic-verifier --> training-example-corpus
  training-example-corpus --> synthetic-diff-generator[Synthetic diff generator]
  synthetic-diff-generator --> diff-verification-pipeline[Diff verification pipeline]
  diff-verification-pipeline --> fine-tuning-dataset[Fine-tuning dataset]
  fine-tuning-dataset --> training-infrastructure[Training infrastructure]
  training-infrastructure --> code-repair-7b-model[Code Repair 7B model]
  code-repair-7b-model --> evaluation-harness[Evaluation harness]

Components

Data flow

The pipeline starts with Replit IDE sessions: operational transformations replay code state, session events and LSP diagnostics identify error moments, and filters remove deterministic or unsuitable cases. The reconstruction pipeline recreates each project filesystem at diagnostic time, serverless lambdas rerun Ruff/Pyright to verify diagnostics, large code LLMs synthesize numbered line diffs with DSPy, verification filters malformed or inapplicable diffs, and the resulting examples fine-tune a 7B DeepSeek-Coder model that is evaluated against public and Replit-specific repair benchmarks.

Why this architecture

Replit chose a specialized, IDE-native fine-tuning pipeline because the platform already had massive diagnostic/event telemetry, but general models were not trained to consume Replit session events or emit a consistently parseable repair format. Starting from real error states kept the input distribution grounded in production usage, synthetic diffs reduced user-data noise, numbered line diffs reduced ambiguity and decoding cost, and a 7B model balanced quality with inference latency and cost.

Models used: DeepSeek-Coder-Instruct-v1.5 · DeepSeek-Coder v1.5 7B Instruct · Starcoder2 · GPT-4 Turbo · GPT-3.5 Turbo · Claude 3 Opus · Claude 3 Haiku · Large pre-trained code LLMs

Scale

• LSP diagnostics are among Replit's most common events, at hundreds of millions per day.
  “This puts LSP diagnostics among our most common events, with hundreds of millions per day.”article, stated
• The initial pipeline targeted 100,000 examples while being designed to scale at least another order of magnitude.
  “We targeted a dataset of 100k examples but designed a pipeline ready to scale up at least another order of magnitude.”article, stated
• The final LeetCode-derived evaluation dataset contained 360 samples.
  “The final distribution of subtypes of problems in our dataset is included in the Appendix and consists of 360 samples.”article, stated
• The Replit-specific repair benchmark contained 389 samples.
  “The final distribution of LSP diagnostic types in our dataset is included in the Appendix and consists of 389 samples.”article, stated
• Training experiments used a single node of 8 H100 GPUs per experiment.
  “We trained on the MosaicML platform with a single node of 8 H100s per experiment.”article, stated
• The data-scaling experiments trained on 10K, 25K, 50K, and 75K subsets.
  “we finetuned DeepSeek-Coder v1.5 7B Instruct on subsets of 10K, 25K, 50K, and 75K training samples.”article, stated

Alternatives considered (roads not taken)

Rely on deterministic LSP CodeActions

Rejected because: Only about 10% of Python LSP diagnostic messages had associated fixes, leaving most repair cases uncovered.

Train from user-fixed errors directly

Rejected because: The source says fixed errors from user data were noisier than synthesized diffs.

Use Unified Diff format

Rejected because: Unified diffs hallucinated line numbers and had higher decoding cost than numbered line diffs.

Add dedicated special tokens for each sentinel token

Rejected because: The team judged architecture/vocabulary changes unnecessary because fine-tuning worked without them and latency gains would be marginal.

Evaluate only on public program-repair datasets

Rejected because: Public datasets may leak into pretraining and are a weak proxy for Replit's real-world user repair distribution.

Patterns

synthetic-data-generation (data-engineering)

Replit generated synthetic numbered line diffs from real diagnostic states instead of relying only on user fixes.

fine-tuning-adaptation (model-customization)

The team fine-tuned a 7B code model to emit Replit's desired line-diff repair format for LSP diagnostics.

structured-output-contract (output-control)

Inputs and outputs are templated into sentinel-token schemas and numbered line diffs so generated repairs can be parsed and applied.

data-quality-loop (data-engineering)

The pipeline verifies reconstructed files, reproduces diagnostics, filters malformed diffs, checks applicability, and uses an LLM filter to improve sample quality.

eval-driven-development (evaluation)

Replit used both a public LeetCode-style benchmark and a custom held-out Replit repair benchmark to evaluate model quality.

latency-optimization (production-optimization)

Model size, diff format, and sentinel-token choices were shaped by inference latency and decoding-cost constraints.

Tradeoffs

Use a specialized 7B model for IDE repair

Chose A fine-tuned 7B DeepSeek-Coder model over Relying only on much larger frontier/API models.

Rationale: The 7B model balanced capability with IDE latency and cost constraints while remaining competitive with larger models on the chosen evaluations.

Synthesize diffs from real error states

Chose Real production error states plus synthetic line diffs over Using user-fixed diffs directly or synthesizing both broken and fixed code end to end.

Rationale: Real inputs preserve production distributions, while synthetic diffs reduce noise and mode collapse risk.

Emit numbered line diffs

Chose Numbered line diffs as the target repair format over Unified diff format.

Rationale: Numbered line diffs reduced ambiguity and decoding cost, while unified diffs had hallucinated line-number behavior in their comparison.

Keep sentinel tokens as ordinary tokenizer pieces

Chose Use textual sentinel tokens mapped to existing tokenizer tokens over Changing the model vocabulary and architecture to add dedicated special tokens.

Rationale: Fine-tuning worked without model surgery, and the expected decoding-latency benefit did not justify changing the architecture.

Evaluate with both public and production-realistic benchmarks

Chose A two-part evaluation: LeetCode-derived repair plus Replit repair over Depending only on academic/public repair datasets.

Rationale: The team needed public comparability and a benchmark that reflected Replit's real inference setting.

Gotchas (what bit them)

LSP diagnostics expose many errors but few deterministic fixes.

Root cause: Diagnostics identify errors, but CodeActions only cover a small fraction of Python diagnostic messages.

Resolution: Use diagnostics as the input signal for a learned repair model, while excluding cases that already have deterministic CodeActions.

Reconstructing executable project state inside Spark was awkward.

Root cause: LSP executables require filesystem directories, while Spark works poorly with dynamically persisted string-backed filesystems.

Resolution: Verify diagnostics with serverless lambdas that can scale up in bursts and provide filesystem-backed execution.

Unified diffs introduced hallucinated line numbers and higher decoding cost.

Root cause: The output format did not constrain the model as effectively for this IDE repair task.

Resolution: Use numbered line diffs, which make application less ambiguous and keep generated sequences shorter.

Generated diffs can be malformed, inapplicable, or incorrect even when produced by strong models.

Root cause: Synthetic repair generation has a large output space and can produce incomplete line diffs or hallucinated content.

Resolution: Layer regex extraction, application checks, and an LLM correctness filter before admitting examples to training.

Public program-repair benchmarks can misrepresent production repair quality.

Root cause: Some benchmark data may appear in model pretraining and often comes from professional repositories rather than Replit's user distribution.

Resolution: Add a held-out Replit repair benchmark sampled at the Repl level and deduplicated to avoid leakage.

Evaluation

Replit evaluated Code Repair 7B with a two-part harness. Public benchmark coverage came from DebugBench-style syntactic/reference-error tasks and newer LeetCode competition problems with synthetic bug injection. Production realism came from a held-out Replit repair benchmark sampled from LSP diagnostic types, deduplicated at the Repl level, and labeled with SOTA LLM proposals plus human annotation. Metrics included functional correctness for executable LeetCode cases and exact-match variants for both public and Replit repair tasks.

Metrics: Functional correctness · AST exact match · AST match with string fallback · Pass@1 · LeetCode repair benchmark performance · Replit repair benchmark performance · Dataset-size scaling · Parameter-count scaling

Outcomes

• Replit Code Repair 7B was competitive with much larger models across the LeetCode and Replit repair benchmarks.
• The real-world Replit repair benchmark remained substantially harder than the LeetCode repair benchmark.
• Replit Code Repair 7B was close to GPT-4 Turbo on the Replit repair benchmark while outperforming most other compared models.
• Performance improved with more training examples in data-scaling experiments.
• Performance improved with larger model sizes in parameter-scaling experiments.
• The team identified future work around cross-file edits, multi-line edits, more programming languages, and post-training with user acceptance/rejection data.

The article reports hundreds of millions of daily LSP diagnostics, a 100K-example initial target, 10K/25K/50K/75K data-scaling experiments, 360 LeetCode repair eval samples, 389 Replit repair eval samples, a single-node 8-H100 training setup, and a 7B model size chosen for production latency and cost.