Building a Low-Latency Global Code Completion Service

github · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Low-latency LLM code completion

GitHub Copilot needed to serve cloud-hosted LLM code completions fast enough to compete with local IDE autocomplete while authenticating users, avoiding wasted inference, and routing traffic globally.

Constraints:

• interactive IDE completions compete with local autocomplete systems that pay no network latency
• TCP and TLS setup costs are large relative to a sub-200-millisecond completion target
• users often keep typing after a request is issued, invalidating the request
• standard HTTP cancellation closes connections and forces expensive reconnects
• global users need nearby model regions, health-based failover, and end-to-end observability

Architecture

GitHub built copilot-proxy, an authenticating HTTP/2 proxy between IDE extensions and Azure-hosted LLMs. IDEs authenticate to GitHub, exchange OAuth credentials for signed short-lived completion tokens, and send completion requests through the proxy. The proxy validates tokens locally, swaps in the provider service key, forwards requests over long-lived HTTP/2 connections, streams completions back, propagates cancellation through request contexts, routes users to nearby proxy/model regions with octoDNS, and relies on GLB/HAProxy for end-to-end HTTP/2 control and graceful redeploys.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  ide-extensions[IDE extensions] --> github-oauth[GitHub OAuth]
  github-oauth --> short-lived-completion-token[short-lived completion token]
  ide-extensions --> glb-load-balancer[GLB load balancer]
  glb-load-balancer --> copilot-proxy[copilot-proxy]
  short-lived-completion-token --> copilot-proxy
  copilot-proxy --> http-2-connection-pool[HTTP/2 connection pool]
  http-2-connection-pool --> azure-hosted-llm-models[Azure-hosted LLM models]
  octodns-routing[octoDNS routing] --> copilot-proxy
  octodns-routing --> azure-hosted-llm-models
  copilot-proxy --> proxy-observability[proxy observability]

Components

Data flow

A developer pauses in an IDE, triggering a completion request. The IDE uses a short-lived code-completion token obtained from GitHub. copilot-proxy validates the token signature, swaps it for the API service key, forwards the request to an Azure-hosted model over pooled HTTP/2 connections, and streams tokens back. If the user types through, the IDE resets the HTTP/2 stream; cancellation propagates through the proxy to the model without closing the underlying connection. octoDNS routes users to healthy nearby proxy/model regions, and proxy metrics define the service SLO.

Why this architecture

GitHub kept an intermediary proxy despite low-latency pressure because it solved authentication, cancellation, observability, regional health, request mutation, traffic splitting, and model-provider abstraction. Colocating proxy and model instances in the same Azure regions made the extra hop cheap enough while preserving operational control.

Models used: Azure-hosted OpenAI models · LLMs for code completion

Scale

• Copilot serves more than 400 million completion requests per day.
  “We serve more than 400 million completion requests. That was when I pitched this talk. I had a look at the number. It's much higher than that these days.”article, stated
• Peak traffic is about 8,000 requests per second.
  “We peak at about 8,000 requests a second during that peak period between the European afternoon and the U.S. work day.”article, stated
• Mean response time is under 200 milliseconds during peak.
  “During that time, we see a mean response time of less than 200 milliseconds. Just in case there is one person who hasn't seen GitHub Copilot in action, here's a recording of me just working on some throwaway code.”article, stated
• GitHub has over 100 million users.
  “We have over 100 million users. We're very proud to call ourselves the home of all developers. The product I'm going to talk to you about is GitHub Copilot, specifically the code completion part of the product.”article, stated
• Around half of issued completion requests are typed-through and unusable.
  “That leads us to the next problem, which is, despite all this work and tuning that went into this, around half of the requests that we issued are what we call typed through. Don't forget, we're doing autocomplete.”article, stated
• About 45% of requests hit the costly closed-socket problem described in the cancellation section.
  “As you saw, in our case, that's about 45% of the requests. Half of the time, we'd be performing that inference and then throwing the result on the floor, which is an enormous waste of money, time, and energy, which in AI terms is all the same thing.”article, stated
• New client releases reach about 80% of the population within 24 to 36 hours, leaving a long tail.
  “When we do a new client release, we'll get to about 80% population within 24 to 36 hours. That last 20% will take until the heat death of the universe.”article, stated
• The proxy team spends about 90% of its time on operations after the core proxy became stable.
  “it gives our small team essentially 90% of our time to get dedicated to the operational issues of running this service.”article, stated

Alternatives considered (roads not taken)

Embed model service keys in the IDE extension

Rejected because: A shipped extension cannot hide a service key from casual or malicious extraction.

HTTP/1-style cancellation by closing connections

Rejected because: Closing the TCP connection forces reconnects for the next immediate completion request and costs more latency than ignoring stale results.

Use standard cloud load balancers/CDNs for HTTP/2

Rejected because: At the time, common providers downgraded backend traffic to HTTP/1 or lacked the HTTP/2 control Copilot needed.

Point-of-presence edge proxy model

Rejected because: Completions always call back to a model, so PoPs created traffic tromboning and operational burden without enough latency benefit.

Have clients connect directly to Azure

Rejected because: Direct Azure connections left authentication and observability unresolved, likely recreating the same proxy layer behind Azure's curtain.

Patterns

latency-optimization (performance)

Copilot uses long-lived HTTP/2 connections, stream-level cancellation, regional colocation, and request streaming to hit IDE-completion latency targets.

cost-aware-routing (operations)

HTTP/2 cancellation prevents paying for stale inference when users type through completion requests.

fallback-degradation (reliability)

Unhealthy regional proxy/model capacity removes itself from DNS so users route elsewhere and receive degraded latency instead of errors.

observability-tracing (operations)

The proxy provides the correct measurement point for latency, cancellation distributions, provider gaps, and end-user SLOs.

model-routing (model_operations)

The proxy can split traffic across model capacity, mirror requests to new model versions, and combine multiple capacity units into a logical model.

canary-rollout (release_management)

Proxy-level traffic splits and mirrored traffic let GitHub test new model versions and experiments without waiting for client rollout.

safety-policy-enforcement (access_control)

The proxy validates signed short-lived tokens and swaps them for the protected provider service key instead of exposing provider credentials to clients.

Tradeoffs

Keep copilot-proxy in the hot path

Chose GitHub-owned authenticating proxy over Direct IDE-to-Azure model calls.

Rationale: The proxy adds a hop, but it solves authentication, observability, request mutation, cancellation, routing, traffic splitting, and operational control.

Use regional proxy/model colocation

Chose Proxy instances colocated with Azure model regions over Many edge PoPs proxying back to a few centralized models.

Rationale: Completions always require model inference, so edge PoPs caused tromboning and operational burden without enough latency benefit.

Invest engineering budget in a bespoke HTTP/2 proxy

Chose Custom proxy accelerating one API call over Pure off-the-shelf glue between model providers and clients.

Rationale: The proxy created a marketable latency and operations advantage where generic infrastructure did not meet requirements.

Gotchas (what bit them)

HTTP cancellation is normally too destructive for rapid autocomplete.

Root cause: HTTP/1 cancellation closes the TCP connection, but autocomplete users usually need another request immediately.

Resolution: Use HTTP/2 streams so each request can be reset while the underlying connection remains open.

End-to-end HTTP/2 support was weaker than expected.

Root cause: Common load balancers and CDNs often spoke HTTP/2 only on the frontend and downgraded backend traffic.

Resolution: Use GitHub's GLB/HAProxy stack to control HTTP/2 end-to-end and preserve client connections across proxy redeployments.

Typed-through completions can waste nearly half of inference work.

Root cause: Users keep typing after the request was sent, so the request context no longer matches the editor state.

Resolution: Propagate cancellation from IDE to proxy to model and add pre-forward checks for immediately canceled requests.

Client rollout is too slow for urgent provider fixes.

Root cause: A long tail of outdated IDE extensions means client-side fixes do not reach everyone quickly.

Resolution: Use the proxy to mutate requests or trigger client upgrade prompts while normal client rollout catches up.

Provider metrics do not describe end-user experience.

Root cause: Model providers optimize their own service view, while users care about end-to-end completion latency.

Resolution: Set SLOs and dashboards at the proxy layer, the data-plane split between client and provider.

Evaluation

GitHub measures Copilot from the proxy/data-plane viewpoint, using latency, success rate, cancellation distributions, regional health, and traffic-split/mirror experiments rather than relying only on client-side averages or provider metrics.

Metrics: mean response time under 200 milliseconds · peak throughput around 8,000 requests per second · typed-through request share around half · success-rate SLO per proxy instance · regional health-check status · cancellation lifetime distributions · traffic split and mirrored model validation

Outcomes

• Copilot serves more than 400 million code-completion requests per day.
• Peak traffic reaches about 8,000 requests per second with mean latency under 200 milliseconds.
• HTTP/2 cancellation avoids wasting roughly half of requests that users type through.
• Regional routing and health checks turn regional failures into reduced capacity rather than total outage.
• The proxy gives GitHub operational levers for request mutation, experiments, model traffic splitting, mirroring, and legacy-client handling.

The source reports over 400 million daily completion requests, about 8,000 requests per second at peak, under-200-millisecond mean response time, over 100 million GitHub users, around half typed-through requests, and global model/proxy regions.