Building Production Evaluation Systems for GitHub Copilot at Scale

github · Tech · 2023 · source Agentic: no · promotional bias: low · summary-only (no fetched source) · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production AI evaluation for developer tools

GitHub needed evaluation systems for Copilot at large production scale, but evaluation tooling and practice had to fit software-engineering workflows, shipping pressure, and implicit user behavior rather than idealized data-science-style offline evals.

Constraints:

• Copilot was serving a massive developer audience, so regressions and quality changes had product and revenue impact.
• The teams building AI products were mostly engineers, while many eval tools assumed data-science workflows.
• Offline evals for LLM products can miss real product behavior and become disconnected from what ships.
• Explicit user feedback signals can be biased and inconsistent.
• Organizational decision-making favored A/B-tested shipping and rapid iteration.

Architecture

The evaluation stack combined code-completion regression harnesses, Microsoft ExP online experimentation, implicit behavioral metrics, executive Shiproom review, and later production-first tracing/logging lessons from Quotient. Offline eval harnesses tested coding changes against repositories and existing tests, while online A/B tests and acceptance/retention metrics became the decisive production signal. The recommended architecture is progressive: ship quickly, instrument from the start, monitor real user behavior, and automate analysis of production traces as systems mature.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  copilot-product-changes["Copilot product changes"] --> code-completion-eval-harness["Code completion eval harness"]
  code-completion-eval-harness --> public-code-repository-corpus["Public code repository corpus"]
  code-completion-eval-harness --> objective-code-test-runner["Objective code test runner"]
  copilot-product-changes --> microsoft-exp-platform["Microsoft ExP platform"]
  microsoft-exp-platform --> implicit-behavior-metrics["Implicit behavior metrics"]
  microsoft-exp-platform --> shiproom-review["Shiproom review"]
  implicit-behavior-metrics --> shiproom-review
  shiproom-review --> copilot-product-changes
  engineering-product-teams["Engineering product teams"] --> production-tracing-sdk["Production tracing SDK"]
  production-tracing-sdk --> trace-and-log-store["Trace and log store"]
  trace-and-log-store --> automated-trace-analysis["Automated trace analysis"]
  automated-trace-analysis --> out-of-box-quality-metrics["Out-of-box quality metrics"]
  out-of-box-quality-metrics --> engineering-product-teams
  offline-eval-platform-prototype["Offline eval platform prototype"] --> engineering-product-teams

Components

Data flow

Model or prompt changes entered an offline code-completion evaluation harness, which generated completions on coding data and checked regressions against objective code signals where possible. Promising changes then flowed through Microsoft ExP A/B tests and weekly Shiproom review, where acceptance rate, edit-distance, and retention-style metrics informed ship decisions. In the later Quotient framing, production traces and logs become the primary data source: teams instrument AI features with a few lines of code, collect user-facing behavior, automate analysis of agent traces, and build evaluation maturity from real production data rather than expecting large offline datasets up front.

Why this architecture

The architecture balances objective code-domain evals with the production reality that successful AI product decisions were driven by online behavior and fast iteration. It keeps offline regression tests where they help, but makes production tracing, A/B testing, implicit usage metrics, and workflow alignment central because those are the signals engineers actually adopt and act on.

Models used: GitHub Copilot models · OpenAI models · Prompt/model combinations

Scale

• The case is framed around Copilot serving 100 million developers.
  “GitHub's experience serving 100 million developers with one of the first major AI-powered developer tools at scale”summary, stated
• The author's Copilot evaluation work took place during 2022-2023.
  “leading the evaluation team at GitHub Copilot during 2022-2023”summary, stated
• Copilot suggestion acceptance rate was around 30% at the time.
  “the acceptance rate of Copilot suggestions, which hovered around 30% at the time.”summary, stated
• Weekly Shiproom review sessions lasted two hours and included GitHub and Microsoft executives.
  “The weekly "Shiproom" meetings, involving executives from both GitHub and Microsoft”summary, stated

  “These two-hour sessions where researchers presented A/B test results for model and prompt changes”summary, stated
• The evaluation-practice discussion spans two years of public 'evals matter' discourse.
  “despite two years of "evals matter" discourse, most AI products are still shipped based on intuition and rapid iteration”summary, stated

Alternatives considered (roads not taken)

Use explicit thumbs-up/down ratings as the primary feedback signal

Rejected because: Explicit feedback introduces bias, self-selection, and inconsistency, so the team favored implicit behavior signals.

Make offline dataset evals the center of LLMOps tooling

Rejected because: Quotient observed engineering teams resisted dataset-focused offline evals and often preferred intuition, custom tooling, or production behavior.

Force engineers into data-science-style evaluation workflows

Rejected because: The summary says most AI product development is done by engineers whose workflows center on CI/CD, monitoring, and alerting.

Require comprehensive evaluation before initial product validation

Rejected because: The recommended approach starts with rapid vibes-based validation and adds monitoring/eval maturity as the product proves users care.

Patterns

eval-driven-development (evaluation)

GitHub built offline code-completion harnesses, A/B testing processes, and production metrics so prompt/model changes could be evaluated before and after release.

canary-rollout (release)

Microsoft ExP A/B testing served as the primary production decision mechanism for Copilot prompt and model changes.

observability-tracing (operations)

The production-first recommendation emphasizes lightweight tracing/logging, automated trace analysis, and visibility into user-facing issues.

data-quality-loop (evaluation)

The recommended strategy collects production data early so teams can understand their AI systems over time and later add more systematic analysis.

llm-as-judge (evaluation)

The later eval-platform framing includes metrics such as similarity, faithfulness, hallucination detection, and tool-use accuracy, which imply automated model-based or metric-based review over traces.

Tradeoffs

Prioritize implicit behavior metrics.

Chose Acceptance rate, edit distance, retained content, and usage behavior over Thumbs up/down and explicit user ratings.

Rationale: Behavioral signals are less exposed to bias and self-selection than explicit feedback mechanisms.

Use online A/B testing as the decisive shipping signal.

Chose ExP-backed online experimentation over Relying only on offline eval harness outcomes.

Rationale: A/B testing reflected real production behavior and became the practical decision mechanism for shipping.

Build production-first evaluation tooling.

Chose Tracing, logging, automated production analysis, and user-facing issue visibility over Dataset-first offline evals as the initial product.

Rationale: Most teams were already testing in production, so tooling needed to meet them there instead of prescribing an idealized workflow.

Start with rapid validation before heavy eval investment.

Chose Vibes-based validation followed by monitoring and systematic eval maturity over Comprehensive up-front evaluation frameworks.

Rationale: The product has to prove that users care before a large evaluation stack is worth slowing development.

Leverage existing enterprise experimentation infrastructure.

Chose Microsoft ExP for production monitoring and A/B tests over Building an isolated evaluation decision stack from scratch.

Rationale: Existing organizational experimentation infrastructure gave the eval system credibility and a route into shipping decisions.

Gotchas (what bit them)

Offline eval tools did not match how engineering teams wanted to build.

Root cause: Evaluation experts designed dataset-centric workflows, while product engineers worked through code, CI/CD, monitoring, and deployment loops.

Resolution: Align eval tooling with existing engineering workflows and production monitoring rather than forcing notebook/spreadsheet practices.

Sophisticated evaluation can still be dominated by shipping pressure.

Root cause: Competitive product environments and A/B-test success can make online experiment results the decisive gate even when broader eval systems exist.

Resolution: Treat production evaluation and monitoring as first-class rather than assuming offline rigor will drive decisions by itself.

Explicit feedback can mislead evaluation.

Root cause: Thumbs up/down and survey-style feedback have bias, self-selection, and inconsistent usage problems.

Resolution: Use implicit behavior signals such as acceptance and retention as more reliable indicators of product utility.

Evaluation rigor degraded as Copilot expanded to more teams.

Root cause: New teams had different skill sets and were primarily engineers rather than data scientists, making the old eval approach harder to maintain uniformly.

Resolution: Build evaluation systems that scale across teams by fitting engineering workflows and automating trace analysis.

Manual review of complex agent traces does not scale.

Root cause: Multi-step AI agents produce increasingly complex trace structures that are too expensive for humans to inspect exhaustively.

Resolution: Automate analysis of production traces and provide out-of-the-box metrics for common failure modes.

Evaluation

The case itself is about evaluation: GitHub used offline coding regression harnesses, ExP A/B testing, implicit behavioral metrics, Shiproom review, and later production-first trace monitoring lessons. The evaluation philosophy is progressive and pragmatic: validate quickly, collect production data from the beginning, then automate deeper analysis as the system and user base mature.

Metrics: Average edit distance between prompts · Average edit distance between LLM responses and retained content · Copilot suggestion acceptance rate · A/B test lift/regression through Microsoft ExP · Code-completion regression tests against public repositories · Hallucination and tool-use accuracy metrics in production traces · User-facing issue visibility from production monitoring

Outcomes

• Copilot evaluation infrastructure combined offline code harnesses with ExP-backed online experimentation.
• A/B tests became the practical shipping gate for model and prompt changes.
• Implicit acceptance and edit-distance metrics became central signals.
• Quotient's offline-eval product direction exposed resistance from engineering teams.
• The case recommends production-first tracing and monitoring that gradually matures into systematic eval.

The case centers on Copilot serving 100 million developers, a 2022-2023 evaluation team, a roughly 30% suggestion acceptance rate, weekly two-hour Shiproom meetings, and later lessons from two years of eval discourse.