Comprehensive LLM Evaluation Framework for Production AI Code Assistants

github · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Evaluation and model selection for production AI coding assistants

GitHub needed to evaluate new Copilot models for performance, quality, safety, latency, and production readiness before exposing them to users across code completion and chat experiences.

Constraints:

Architecture

GitHub evaluates Copilot models with a layered offline and internal-live evaluation pipeline. More than 4,000 offline tests run mostly in automated CI, including about 100 containerized repositories where models must repair deliberately failing code. Copilot Chat is evaluated with more than 1,000 technical questions and an audited LLM-as-judge for complex answers. A proxy server lets GitHub swap model APIs without client changes, GitHub Actions runs tests, Apache Kafka and Microsoft Azure move/results-store data, dashboards support analysis, and daily production-model tests detect quality degradation.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  model-candidates["model candidates"] --> copilot-proxy-server["Copilot proxy server"]
  copilot-proxy-server --> code-completion-feature["code completion feature"]
  model-candidates --> offline-test-platform["offline test platform"]
  offline-test-platform --> containerized-repository-suite["containerized repository suite"]
  containerized-repository-suite --> ci-test-batteries["CI test batteries"]
  model-candidates --> copilot-chat-question-set["Copilot Chat question set"]
  copilot-chat-question-set --> llm-judge["LLM judge"]
  llm-judge --> judge-audit-process["judge audit process"]
  offline-test-platform --> apache-kafka-pipeline["Apache Kafka pipeline"]
  apache-kafka-pipeline --> azure-data-systems["Azure data systems"]
  azure-data-systems --> evaluation-dashboards["evaluation dashboards"]
  judge-audit-process --> evaluation-dashboards
  evaluation-dashboards --> model-adoption-decision["model adoption decision"]
  model-adoption-decision --> model-candidates
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
model candidates Provider models evaluated for Copilot quality, performance, and safety. Copilot proxy server, offline test platform, Copilot Chat question set
Copilot proxy server Routes code-completion requests to different model APIs without client-side changes. model candidates, code completion feature
code completion feature Copilot product surface using the proxy for model-backed code suggestions. Copilot proxy server
offline test platform Custom evaluation platform built primarily with GitHub Actions. model candidates, containerized repository suite, Apache Kafka pipeline
containerized repository suite Around 100 repositories with passing CI that are deliberately broken for model repair tests. offline test platform, CI test batteries
CI test batteries Automated tests that determine whether a candidate model restored code to a passing state. containerized repository suite, offline test platform
Copilot Chat question set More than 1,000 technical questions used to evaluate chat answer quality. model candidates, LLM judge
LLM judge Known-good model used to evaluate complex chat answers at scale. Copilot Chat question set, judge audit process
judge audit process Routine auditing to ensure the evaluation LLM aligns with human reviewers and behaves consistently. LLM judge, evaluation dashboards
Apache Kafka pipeline Moves evaluation results between systems. offline test platform, Azure data systems
Azure data systems Stores or processes evaluation results for analysis. Apache Kafka pipeline, evaluation dashboards
evaluation dashboards Exploration layer for quality, performance, safety, and production-monitoring data. Azure data systems, judge audit process, model adoption decision
model adoption decision Human decision process weighing metrics, latency, acceptance, and responsible AI standards. evaluation dashboards, model candidates

Data flow

A candidate model is routed through Copilot's proxy without changing product code, then evaluated on offline code-repair scenarios and chat questions. Repository tests modify known-good containerized projects into failing states and check whether the model restores passing CI; chat tests compare answers to technical questions, sometimes using an audited judge model. Test results flow through GitHub Actions, Kafka, Azure, and dashboards; production models are retested daily and audited when degradation appears.

3 supporting quotes
“We run more than 4,000 offline tests, most of them as part of our automated CI pipeline. We also conduct live internal evaluations, similar to canary testing, where we switch a number of Hubbers to use a new model. We test all major changes to GitHub Copilot”article, stated
“We have a proxy server built into our infrastructure that the code completion feature uses. We can easily change which API the proxy server calls for responses without any sort of change on the client side. This lets us rapidly iterate on new models without”article, stated
“Results are then piped in and out of systems like Apache Kafka and Microsoft Azure, and we leverage a variety of dashboards to explore the data.”article, stated

Why this architecture

GitHub combines automated tests, LLM judging, internal canaries, proxy-based model swapping, dashboards, and daily production checks because Copilot must support rapid model iteration while preserving quality, safety, and user trust.

3 supporting quotes
“frameworks and benchmarking tools available, but we generally build our own evaluation tools. Automated tests enable evaluation at scale. You can see how well a model performs on a large number of tasks. The downside is that you tend to need some objective criteria”article, stated
“Combining the two approaches enables you to get a sense of the subjective quality of responses without relying entirely on anecdotal evidence”article, stated
“This lets us rapidly iterate on new models without having to change any of the product code. All of these tests run on our own custom platform built primarily with GitHub Actions. Results are then piped in and out of systems like Apache Kafka and Microsoft”article, stated

Models used: Claude 3.5 Sonnet · Gemini 1.5 Pro · o1-preview · o1-mini · OpenAI models · Claude models · Gemini models · Llama models · known-good LLM judge

Scale

Alternatives considered (roads not taken)

Adopt the latest model because it is newer

Rejected because: The article warns that newer models may not perform better for a specific use case.

“It’s tempting to hop immediately to the latest models. But just because a model is newer doesn’t mean it will perform better for your use case.”article, stated

Rely only on automated tests

Rejected because: Automated tests scale but require objective criteria and miss subjective output quality.

“well a model performs on a large number of tasks. The downside is that you tend to need some objective criteria to evaluate the output. Manual testing enables more subjective evaluations of output quality and accuracy, but is more time intensive. Combining the two approaches enables you to get a”article, stated
“Manual testing enables more subjective evaluations of output quality and accuracy, but is more time intensive.”article, stated

Manual evaluation of thousands of chat responses

Rejected because: GitHub uses another LLM to scale evaluation beyond what manual testing can accomplish.

“we use another LLM to check the answers provided by the model we’re evaluating to scale our efforts beyond what we could accomplish through manual testing.”article, stated

Change product code for each model experiment

Rejected because: The proxy server can change the API called for responses without client-side changes.

“We can easily change which API the proxy server calls for responses without any sort of change on the client side.”article, stated

Simple threshold-only model adoption

Rejected because: Some adoption decisions involve tradeoffs such as higher acceptance rates with higher latency.

“what if a model shows a substantial bump in acceptance rates, but also increases latency? There are sometimes inverse relationships between metrics. Higher latency might actually lead to a higher acceptance rate because users might see fewer suggestions”article, stated
“what if a model shows a substantial bump in acceptance rates, but also increases latency? There are sometimes inverse relationships between metrics. Higher latency might actually lead to a higher acceptance rate because users might see fewer suggestions total. GitHub’s”article, stated

Patterns

eval-driven-development (evaluation)

GitHub gates Copilot changes through thousands of offline tests, repository repair tasks, chat-question tests, internal live evaluations, and daily production checks.

“We test all major changes to GitHub Copilot this way, not just potential new models.”article, stated
“We also run these tests against our production models every day.”article, stated

llm-as-judge (evaluation)

GitHub uses a known-good LLM to evaluate complex Copilot Chat answers and audits judge outputs for alignment and consistency.

“we use another LLM to check the answers provided by the model we’re evaluating”article, stated
“We also routinely audit the outputs of this LLM in evaluation scenarios to make sure it’s working correctly.”article, stated

canary-rollout (release-safety)

GitHub conducts internal live evaluations by switching some Hubbers to a new model before broader rollout.

“We also conduct live internal evaluations, similar to canary testing, where we switch a number of Hubbers to use a new model.”article, stated

model-routing (infrastructure)

The proxy server can route Copilot code completion to different model APIs without client-side product changes.

“We can easily change which API the proxy server calls for responses without any sort of change on the client side.”article, stated

observability-tracing (operations)

Evaluation results flow into dashboards, and daily production checks trigger audits when degradation is detected.

“Results are then piped in and out of systems like Apache Kafka and Microsoft Azure, and we leverage a variety of dashboards to explore the data. ## Making the call to adopt or not The big question is what to do with the data we’ve collected.”article, stated
“If we see degradation, we do some auditing to find out why the models aren’t performing as well as they used to.”article, stated

safety-policy-enforcement (responsible-ai)

The source says model evaluation covers performance, quality, and safety before production changes.

“While adding models to GitHub Copilot, performance, quality, and safety was always kept top of mind.”article, stated

Tradeoffs

Use both automated and manual/subjective evaluation.

Chose Hybrid automated tests plus manual or judge-based assessment over Only automated or only manual evaluation.

Rationale: Automated tests scale to thousands of scenarios, while manual or model-judged checks capture subjective quality without relying only on anecdotes.

“Combining the two approaches enables you to get a sense of the subjective quality of responses without relying entirely on anecdotal evidence”article, stated

Use containerized repository repair as a code-quality benchmark.

Chose Break known-good repos and ask the model to restore passing tests over Only static code-generation benchmarks.

Rationale: Real repositories with CI failures approximate developer workflows better than isolated prompts.

“We modify those repositories to fail the tests, and then see whether the model can modify the codebase to once again pass the failing tests.”article, stated

Put model switching behind a proxy.

Chose Server-side proxy API routing over Client-side product code changes for every model test.

Rationale: The proxy accelerates model iteration without changing product code.

“This lets us rapidly iterate on new models without having to change any of the product code.”article, stated

Audit production models daily.

Chose Continuous tests and prompt/configuration fixes over One-time prelaunch model evaluation.

Rationale: Daily production checks catch degradation and prompt changes can restore expected quality.

“We also run these tests against our production models every day. If we see degradation, we do some auditing to find out why the models aren’t performing as well as they used to.”article, stated
“Sometimes we need to make changes, for example modify some prompts, to get back to the quality level we expect.”article, stated

Gotchas (what bit them)

Automated tests alone miss subjective answer quality.

Root cause: Automated tests need objective criteria, while chat answers can require nuanced quality and accuracy judgment.

Resolution: Combine automated tests with manual evaluation and an audited LLM-as-judge for complex chat answers.

“Automated tests enable evaluation at scale. You can see how well a model performs on a large number of tasks. The downside is that you tend to need some objective criteria to evaluate the output.”article, stated
“Manual testing enables more subjective evaluations of output quality and accuracy, but is more time intensive.”article, stated

LLM-as-judge evaluation can drift from human judgment.

Root cause: The judge model must stay aligned with human reviewers and consistent across many requests.

Resolution: Use a model with known good performance and routinely audit its evaluation outputs.

“It’s a challenge to ensure that the LLM we use for evaluating answers are aligned with human reviewers and perform consistently over many requests.”article, stated
“We also routinely audit the outputs of this LLM in evaluation scenarios to make sure it’s working correctly.”article, stated

Model quality can degrade after production launch.

Root cause: Production model behavior can change or drift, requiring daily checks and audits.

Resolution: Run tests against production models every day and adjust prompts or other configuration when quality drops.

“If we see degradation, we do some auditing to find out why the models aren’t performing as well as they used to.”article, stated
“Sometimes we need to make changes, for example modify some prompts, to get back to the quality level we expect.”article, stated

Latency and acceptance metrics can move in opposite directions.

Root cause: Higher latency may reduce the number of suggestions users see, which can distort acceptance rate interpretation.

Resolution: Analyze metric relationships and use human judgment rather than simple thresholds for adoption decisions.

“what if a model shows a substantial bump in acceptance rates, but also increases latency?”article, stated
“Higher latency might actually lead to a higher acceptance rate because users might see fewer suggestions total.”article, stated

Repository repair tests are expensive to maintain.

Root cause: Realistic coverage requires many scenarios, languages, frameworks, and language versions.

Resolution: Continuously expand the containerized repository test suite despite the time investment.

“We generate as many different scenarios as we can in different programming languages and frameworks.”article, stated
“This work takes a lot of time, but it’s the best way we have to evaluate a model’s quality.”article, stated

Evaluation

GitHub evaluates Copilot changes through over 4,000 offline tests, around 100 broken-repository repair scenarios, more than 1,000 technical chat questions, LLM-as-judge checks with audits, internal canary-style live evaluations, dashboards, token usage analysis, latency/acceptance tradeoff analysis, and daily production-model testing.

Metrics: unit test pass rate · similarity to original passing state · percentage of chat questions answered correctly · token usage · latency · acceptance rates · production degradation signals · judge-model audit quality

3 supporting quotes
“## Automating code quality tests We run more than 4,000 offline tests, most of them as part of our automated CI pipeline.”article, stated
“We have a collection of more than 1,000 technical questions we use to evaluate the quality of a model’s chat capabilities.”article, stated
“All of these tests run on our own custom platform built primarily with GitHub Actions.”article, stated

Outcomes

The source reports more than 4,000 offline tests, around 100 containerized repositories, more than 1,000 technical chat questions, daily production-model tests, and internal live evaluations of selected Hubbers before broader rollout.

3 supporting quotes
“GitHub’s goal is to create the best quality, most responsible AI coding assistant possible”article, stated
“we couldn’t make them without the quality data our evaluation procedures provide.”article, stated
“Hopefully we’ve given you a few ideas you can apply to your own use cases.”article, stated