Building Production-Scale Code Completion Tools with Continuous Evaluation and Prompt Engineering

gitlab · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production code-completion evaluation and prompt engineering

GitLab needed to build code completion tooling that helps developers write useful code while continuously evaluating multiple LLMs, prompt templates, and fine-tuning options at production scale rather than relying on manual prompt spreadsheets.

Constraints:

• Code completion suggestions must be honest, harmless, and helpful to developers.
• Developer experience depends on latency, cost, and usable suggestions.
• Third-party and open-source LLM training data may be unavailable or opaque.
• Quality is hard to judge without evaluating at scale against actual code and developer patterns.
• Prompt changes and model choices need version control, CI/CD, and continuous feedback.
• Code completion usage continuously creates new data that can feed evaluation and training loops.

Architecture

GitLab's Model Ops architecture for GitLab Suggestions combines a multi-model completion gateway with a continuous evaluation loop. Historical code and completion data are tokenized, compared against developer commit patterns, and used to define agreeable developer outputs. A prompt engine with a prompt library/DB processes code and prompt templates; a gateway routes requests to third-party or internally tuned models; prompt validators and rate limiters score outputs; CI/CD-connected microservices run evaluation continuously; and low-quality outputs can trigger prompt tuning or fine-tuning of open-source models through a version-controlled loop.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  developer[Developer] --> gitlab-suggestions[GitLab Suggestions]
  gitlab-suggestions --> gateway[Gateway]
  gateway --> prompt-engine[Prompt engine]
  prompt-engine --> prompt-library[Prompt library]
  prompt-engine --> tokenizer[Tokenizer]
  tokenizer --> historical-code-database[Historical code database]
  historical-code-database --> scoring-engine[Scoring engine]
  prompt-engine --> scoring-engine
  scoring-engine --> evaluation-microservices[Evaluation microservices]
  evaluation-microservices --> ci-cd[CI/CD]
  ci-cd --> prompt-library
  gateway --> prompt-validator[Prompt validator]
  prompt-validator --> rate-limiter[Rate limiter]
  prompt-validator --> fine-tuning-loop[Fine-tuning loop]
  gateway --> model-pool[Model pool]
  model-pool --> training-and-tuning-environment[Training and tuning environment]
  training-and-tuning-environment --> checkpoint-layers[Checkpoint layers]
  fine-tuning-loop --> training-and-tuning-environment
  completion-data-stream[Completion data stream] --> historical-code-database
  completion-data-stream --> evaluation-microservices
  completion-data-stream --> fine-tuning-loop

Components

Data flow

A developer requests a completion through GitLab Suggestions. The gateway invokes the prompt engine, which uses the prompt library and tokenization over current and historical code. Historical commits and similarity algorithms define an agreeable developer output. Candidate outputs from open-source, tuned, or third-party LLMs are compared against that benchmark by evaluation microservices. Prompt validators, rate limiters, CI/CD, and version control feed improvements back into prompt templates or fine-tuning loops, while new completion data continuously refreshes the evaluation and training corpus.

Why this architecture

GitLab's architecture exists because model quality cannot be assessed by intuition or one-off prompt inspection. Multiple raw models, opaque training data, latency/cost constraints, and developer-specific code patterns require a continuous benchmark loop that compares LLM output to historical developer behavior, version-controls prompt and model changes, and escalates from prompt tuning to fine-tuning when needed.

Models used: Open-source pre-trained LLMs · Third-party LLMs · Tuned LLMs · ChatGPT · XGBoost

Scale

• The team had been building code completions for about six to seven months.
  “we started this journey on building code completions through uh through two cockle suggestions uh pretty much around four or six seven months”transcript, stated
• The desired developer productivity framing is moving from 500 lines to 1,000 lines of usable feature code.
  “instead of writing 500 lines can actually write thousand lines of code for a usable feature”transcript, stated
• The architecture is intended to run evaluation thousands of times in continuous background fashion.
  “this is something that can help just doing it at scale thousand times for in that continuous background fashion”transcript, stated
• A raw LLM combination is described as potentially starting at a 10% coder acceptance rate.
  “raw llms of a combination giving you a 10 acceptance rate for coders”transcript, stated

Alternatives considered (roads not taken)

Pick a single LLM

Rejected because: The presenter argues production code completion involves choosing many models based on objective, training data, benchmarks, weights, cost, and latency.

Manual spreadsheet prompt evaluation

Rejected because: Spreadsheets cannot continuously evaluate outputs at scale or feed an instant improvement loop.

Prompt tuning only

Rejected because: When a prompt validator cannot reach output quality, the system should route to model fine-tuning.

Trust user-facing model output without historical comparison

Rejected because: The system needs an agreeable developer output baseline derived from commits and similarity.

Patterns

eval-driven-development (evaluation)

GitLab's architecture continuously evaluates code-completion outputs against historical developer behavior and feeds the results into prompt/model improvement.

prompt-versioning (prompt-operations)

The prompt library/DB, prompt validator, version-based control, and CI/CD path make prompt templates tracked deployable artifacts.

model-routing (model-operations)

The gateway can call different third-party or pre-trained models depending on user input and quality needs.

fine-tuning-adaptation (model-customization)

Open-source model weights can be tuned with additional data and fine-tuned when prompt validation is not enough.

data-quality-loop (data-engineering)

Continuous completion data and historical commit data become the baseline for ranking outputs and improving prompts/training.

cost-aware-routing (production-optimization)

Model choice explicitly includes cost and latency alongside objective, training data, benchmarks, and weights.

Tradeoffs

Use many model options

Chose Open-source, tuned, and third-party LLMs behind a routing gateway over A single fixed completion model.

Rationale: Different models offer different weights, tuning options, costs, latency, and benchmark behavior for code-completion use cases.

Evaluate against agreeable developer output

Chose Historical commit and similarity-derived benchmarks over Pure generic benchmark or manual reviewer judgment.

Rationale: The system needs a GitLab-specific proxy for what helpful developer code should look like.

Automate prompt evaluation

Chose Microservice engines running in continuous loops over Manual spreadsheet review.

Rationale: Continuous service-based evaluation can run at scale and keep improving completion usefulness without manual bottlenecks.

Escalate from prompt tuning to fine-tuning

Chose Prompt templates first, fine-tuning when validation cannot meet quality over Only retraining models or only editing prompts.

Rationale: Prompt changes are fast, but the architecture keeps a deeper model-adaptation path for failures that need it.

Gotchas (what bit them)

Quality is hard to assess without scaled evaluation.

Root cause: Raw model benchmarks and training-data descriptions do not reveal whether completions help GitLab developers in context.

Resolution: Build scaled evaluation over historical code, prompt outputs, and similarity to agreeable developer outputs.

Opaque model training data complicates quality judgment.

Root cause: For many third-party or open-source LLMs, the team cannot constantly inspect or feed the training data used to judge quality.

Resolution: Evaluate outputs against GitLab-specific historical developer patterns instead of relying only on model provenance.

Manual prompt review does not scale.

Root cause: Spreadsheet-based prompt evaluation cannot run continuously or instantaneously across many prompt/model outputs.

Resolution: Implement microservice evaluation engines that loop continuously and feed CI/CD-controlled prompt and model changes.

Prompt improvements may not be enough.

Root cause: Some completion failures cannot be repaired by instantaneous prompt-template tuning.

Resolution: Route unresolved quality gaps into fine-tuning through the continuous loop.

The useful benchmark is developer intent, not only syntactic correctness.

Root cause: Code completion must be honest, harmless, and helpful, with helpfulness tied to the coder's actual goal.

Resolution: Frame evaluation around developer usefulness and compare suggestions to what developers actually committed.

Evaluation

GitLab evaluates code-completion usefulness by processing prompt input, tokenizing code, comparing candidate outputs to historically committed developer code, scoring similarity to an agreeable developer output, and running this as continuous microservice engines connected to version control and CI/CD. The framework measures whether suggestions are honest, harmless, and helpful, with particular focus on helpfulness for developer goals.

Metrics: Honest/harmless/helpful dimensions · Developer acceptance rate · Similarity to agreeable developer output · Prompt validator rating · Historical commit similarity · Prompt output quality · Cost · Latency · Benchmark performance · Continuous CI/CD evaluation results

Outcomes

• GitLab built a continuous evaluation and prompt-engineering architecture for GitLab Suggestions over a roughly six-to-seven-month journey.
• The proposed architecture moves prompt evaluation from manual spreadsheets into background microservice engines.
• The system defines agreeable developer output from historical commits and similarity, then compares model outputs against that target.
• Prompt validators, rate limiters, version control, CI/CD, and fine-tuning loops support ongoing improvement.
• The talk frames a raw LLM baseline around 10% coder acceptance and describes continuous evaluation/prompting as the path to dialing up usability.
• The speaker emphasizes that continuous code data remains central even in the age of LLMs.

The transcript quantifies the build as roughly four/six/seven months, frames productivity as moving from 500 to 1,000 lines of usable feature code, describes evaluation at thousand-times continuous background scale, and mentions a raw LLM combination starting around a 10 acceptance rate for coders.