Building Robust Evaluation Systems for GitHub Copilot

github · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Evaluation infrastructure for AI coding assistants

GitHub needed evaluation systems that could guide Copilot from code-completion experiments into production releases and later chat/tool-use capabilities, where correctness ranged from executable code to subjective conversation quality.

Constraints:

• Generated code cannot always be judged by looking at the output; many cases require running code or tests.
• Offline evaluations had to avoid training-data leakage while still approximating production traffic.
• Production rollout decisions needed a small set of interpretable metrics plus diagnostic guardrails.
• Chat interactions often had no executable output, so execution-based completion evals were insufficient.
• Human side-by-side evaluation was too slow and expensive to scale.
• Tool-use evaluation had to distinguish overlapping tool choices rather than just judge final text.

Architecture

GitHub Copilot used a layered evaluation architecture. Harness lib built offline verifiable code-completion tests from open-source Python and JavaScript repositories whose tests passed, mapped functions to unit tests with coverage, removed candidate implementations, asked Copilot to regenerate them, and reran tests. Ship-room A/B experiments then ramped online changes around 10% traffic while watching completion acceptance, characters retained, latency, and many guardrails. Copilot Chat added Label Studio human comparisons and then GPT-4 LLM-as-judge prompts with granular criteria. Tool-use evaluations checked whether models selected the correct function/tool calls and used confusion matrices to expose routing overlap.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  open-source-repository-sampler[Open source repository sampler] --> training-cutoff-filter[Training cutoff filter]
  open-source-repository-sampler --> coverage-mapper[Coverage mapper]
  training-cutoff-filter --> candidate-function-set[Candidate function set]
  coverage-mapper --> candidate-function-set
  candidate-function-set --> harness-lib[Harness lib]
  harness-lib --> copilot-completion-model[Copilot completion model]
  harness-lib --> unit-test-runner[Unit test runner]
  unit-test-runner --> offline-scorecard[Offline scorecard]
  harness-lib --> offline-scorecard
  offline-scorecard --> ship-room-a-b-experiment[Ship room A/B experiment]
  copilot-completion-model --> ship-room-a-b-experiment[Ship room A/B experiment]
  ship-room-a-b-experiment --> production-metric-dashboard[Production metric dashboard]
  production-metric-dashboard --> data-science-investigation[Data science investigation]
  data-science-investigation --> ship-room-a-b-experiment
  copilot-chat-prototype[Copilot Chat prototype] --> label-studio-human-comparison[Label Studio human comparison]
  copilot-chat-prototype --> llm-judge-prompt[LLM judge prompt]
  label-studio-human-comparison --> llm-judge-prompt
  tool-call-validator[Tool call validator] --> confusion-matrix[Confusion matrix]

Components

Data flow

For offline completion evaluation, GitHub selected Python and JavaScript repositories, kept repos whose tests passed, used coverage tools to connect functions to unit tests, filtered candidate functions, removed the implementation body, let Copilot regenerate the body, and reran the unit tests to measure directional improvement. For online releases, promising variants entered A/B tests at around 10% traffic and were judged by acceptance rate, characters retained, latency, and guardrail metrics. For chat, human side-by-sides in Label Studio seeded a granular GPT-4 judge rubric. For tool use, expected versus actual tool calls fed confusion matrices to reveal overlapping tools.

Why this architecture

The architecture deliberately used multiple evaluation modes because Copilot's product surfaces had different observability properties. Code completion could be verified by execution, production launches needed user-behavior metrics, chat required subjective conversation judgments, and tool use needed exact or diagnostic checks on function selection. The team also kept experimental notebook-style evaluation separate from repeatable regression and ship-room pipelines because model/product discovery and launch decisions had different reliability needs.

Models used: GitHub Copilot code-completion models · Copilot Chat models · GPT-4 as LLM judge

Scale

• GitHub narrowed offline harness repositories to Python and JavaScript because those tests were practical to run.
  “Specifically, in our case, Python and JavaScript. Uh we had to narrow it down. Um you run their test. That's basically why we had to narrow it down.”transcript, stated
• Candidate functions needed at least one unit test and coverage before entering the evaluation set.
  “the candidate functions had to have one unit test uh had to with at least some percentage of its lines covered.”transcript, stated
• Early Copilot baselines were around 40% to 50% success in the harness by the speaker's recollection.
  “as a baseline we know what version X of of copilot did. It wouldn't succeed at all and I forgot what it was up to. It was like 40 or 50%.”transcript, stated
• Online rollout started around 10% traffic before ramping up.
  “we'll start them start our uh users on like 10% of the traffic and kind of wrap it up.”transcript, stated
• The online ship-room process watched high tens to low hundreds of guardrail metrics beyond the key metrics.
  “we had several guardrail metrics. Uh and by several, I mean high tens, maybe maybe even in low low in the the 100.”transcript, stated
• Investigating an alarming production metric could delay launch by a week.
  “this is concerning enough that it's worth, you know, delaying the launch for a week while someone digs into the logs”transcript, stated
• The web Copilot interface had roughly five or six tools when the speaker left, later about double that.
  “I think there was probably when I was there, just a handful, like five, six. I think since then, it's kind of moved up to about about double that”transcript, stated

Alternatives considered (roads not taken)

Judge generated code only by inspecting the text

Rejected because: Copilot completions needed verifiable evaluation: generated code could not be immediately assessed by sight, but it could be compiled or tested.

Use the existing product as the only evaluation harness

Rejected because: New model capabilities and unlaunched product ideas required experimental harnesses that could prototype uses before the product existed.

Rely only on human side-by-side judgment for Copilot Chat

Rejected because: Human evaluation could capture subjective quality, but it was slow, expensive, and guideline-heavy.

Ask a vague LLM judge which answer is better

Rejected because: Early GPT-4 judgments drifted to irrelevant criteria, so GitHub decomposed the rubric into explicit checklist items.

Optimize only for acceptance rate

Rejected because: Acceptance could be inflated by tiny or poor completions, so characters retained, latency, and guardrails had to be interpreted together.

Patterns

eval-driven-development (quality)

Copilot development used offline harnesses, A/B tests, LLM judges, and algorithmic tool-use checks to decide whether model, prompt, context, or product changes were safe to ship.

llm-as-judge (evaluation)

For Copilot Chat, GitHub turned vague human preference work into granular GPT-4 judging criteria that were quick for humans to verify.

structured-output-contract (validation)

The evaluation taxonomy separated simple algorithmic checks such as JSON/schema adherence and response length from more complex verifiable or subjective evaluations.

tool-use-orchestration (architecture)

GitHub evaluated function-calling behavior by comparing expected and actual tool calls, then used confusion matrices to reveal when tools overlapped too much.

canary-rollout (release)

Copilot shipped changes through A/B experiments that began at a small traffic percentage and ramped while watching key and guardrail metrics.

Tradeoffs

Evaluate completions by execution

Chose Verifiable unit-test harness over Pure text inspection or simple equality checks.

Rationale: Code completion quality has degrees of freedom: generated code may differ textually while still compiling and passing tests.

Use production A/B metrics after offline tests

Chose Ship-room traffic ramp with key and guardrail metrics over Relying on offline harness results alone.

Rationale: Offline tests reduced surprise but could not prove the change behaved appropriately for real users.

Granular LLM judging for chat

Chose Detailed GPT-4 checklist criteria over Open-ended human or LLM preference prompts.

Rationale: Specific checklists made the judge reliable enough to review quickly and avoided irrelevant stylistic criteria.

Keep separate evaluation styles

Chose Flexible notebooks and modular harness blocks plus canned scripts/pipelines over A single rigid evaluation pipeline.

Rationale: Researchers needed to prototype new uses before a product existed, while shippers needed reliable repeatable numbers.

Shorten some completions

Chose Incremental suggestions users can tab through over One-shot long code completions.

Rationale: Users sometimes accepted long completions reluctantly and then deleted half; shorter completions better matched incremental coding behavior.

Gotchas (what bit them)

Evaluation repositories can leak training data or misrepresent production traffic.

Root cause: GitHub had to choose repositories outside the model training set, which biased the harness toward newer and smaller codebases.

Resolution: Use training-data cutoff information where available and treat repository representativeness as a known limitation.

Optimizing a single production metric can degrade the user experience.

Root cause: Acceptance rate alone can favor many tiny completions, while users may accept and then heavily edit poor suggestions.

Resolution: Use retained characters, latency, and guardrail metrics as diagnostic companions to acceptance rate.

A vague LLM judge may reward irrelevant features.

Root cause: Early model judges followed broad prompts poorly and latched onto stylistic details unrelated to the intended criterion.

Resolution: Micromanage the judge with explicit checklist criteria for the behavior being evaluated.

One evaluation pipeline cannot serve both experimental design and shipping decisions equally well.

Root cause: Exploratory product/model work needs modular notebooks and prototypes, while release gates need repeatable scripted metrics.

Resolution: Support both experimental Lego-block workflows and canned regression/ship-room pipelines.

Tools that overlap semantically confuse the orchestrator model.

Root cause: When two tools cover similar operating space, expected and actual tool calls diverge.

Resolution: Use confusion matrices to identify overlapping tool definitions and make tools partition the space more clearly.

Evaluation

GitHub used a four-part evaluation stack: verifiable offline harness tests for code completion, A/B testing for online rollout, LLM-as-judge for subjective chat quality, and algorithmic tool-call validation for function-calling behavior.

Metrics: Offline unit-test pass rate for regenerated functions · Completion acceptance rate · Characters retained · Latency · High tens to low hundreds of guardrail metrics · Human/LLM side-by-side conversation preference · Tool-call exact match and confusion matrix behavior

Outcomes

• The evaluation system helped Copilot test model, prompt, context, and tokenization changes before and during production rollout.
• Granular LLM-as-judge rubrics made subjective Copilot Chat evaluation more reliable than vague preference prompts.
• A/B guardrails exposed metric coupling and sometimes delayed launches until teams understood anomalies.
• Confusion matrices gave the tool-use team a way to diagnose overlapping tool definitions.

Copilot's evaluation architecture evolved across offline completion tests, ship-room A/B experiments, chat-quality judging, and tool-use checks, with production rollout beginning around 10% traffic and monitored by a few key metrics plus high tens to low hundreds of guardrails.