LLM-Driven Developer Experience and Code Migrations at Scale

uber · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Developer productivity and code migration

Uber needed to improve developer experience and reduce technical debt across massive monorepos without simply adding headcount, while avoiding production risk from unreliable generated code.

Constraints:

• Uber's six monorepos together contain more than 100 million lines of code.
• Developer tools had to integrate across Xcode, JetBrains IDEs, VS Code, and multiple language workflows.
• Autocomplete experiences required very low latency, with a target below one second for 100 tokens.
• Mobile migration automation carried high rollout risk because hallucinated code could cause production outages and mobile releases roll out conservatively.

Architecture

Uber's developer-platform strategy evolved into three tracks: build on GitHub Copilot rather than maintain a competing in-house code assistant, use agentic workflows such as AutoCover to generate and validate tests with developers in the loop, and accelerate Java-to-Kotlin migration by combining deterministic AST tooling with LLM-authored migration rules reviewed by platform engineers. A horizontal AI Developer Experience team works with monorepo experts, while ML infrastructure provides hosting, fine-tuning, and gateway capabilities.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  AI_Developer_Experience_team["AI Developer Experience team"] --> Developer_IDE_surfaces["Developer IDE surfaces"]
  ML_infrastructure_team["ML infrastructure team"] --> AI_Developer_Experience_team
  Developer_IDE_surfaces --> GitHub_Copilot["GitHub Copilot"]
  GitHub_Copilot --> AutoCover["AutoCover"]
  AutoCover --> Build_and_test_runner["Build and test runner"]
  Build_and_test_runner --> AutoCover
  Java_to_Kotlin_migration_system["Java to Kotlin migration system"] --> Historical_migration_dataset["Historical migration dataset"]
  Historical_migration_dataset --> Rule_writing_agent["Rule-writing agent"]
  Rule_writing_agent --> CI_validation["CI validation"]
  CI_validation --> Rule_writing_agent
  Rule_writing_agent --> Platform_engineer_review["Platform engineer review"]
  Platform_engineer_review --> Java_to_Kotlin_migration_system

Components

Data flow

For the paused in-house assistant, local code context was gathered in the IDE, sent through proxies to fine-tuned backend models, verified for syntax, and displayed to developers. For AutoCover, a workflow prepares mocks and files, generates tests with an LLM, builds and runs tests, loops through LLM fixes, validates assertions, and refactors duplicates. For Java-to-Kotlin migration, historical migration deltas are mined, an agent drafts AST rules, CI runs conversion/build/test/lint, failures are fed back to another LLM, and successful deterministic rules go to platform review.

Why this architecture

Uber learned that building a full internal assistant would keep them in a costly catch-up race, so they built on Copilot for IDE surfaces and reserved internal AI investment for workflow-specific automation where Uber context and deterministic validation mattered.

Models used: GitHub Copilot · Starcoder · Code Llama · fine-tuned code models · LangChain · LangGraph · LLM repair loop · LLM-authored AST rules

Scale

• Uber's combined monorepos exceed 100 million lines of code.
  “there would be over 100 million lines of code.”transcript, stated
• Uber's Android monorepo has well over 10 million lines and close to 100,000 files across Java and Kotlin.
  “Well over 10 million lines of code total between the two, and close to 100,000 files between the two.”transcript, stated
• The internal assistant target was less than one second for 100 tokens.
  “we could get latency down to less than one second for 100 tokens.”transcript, stated
• Copilot had about 60% active 30-day developer usage at Uber.
  “today we have something around 60% of our developers that are active 30 day users for CoPilot.”transcript, stated
• Copilot users showed roughly a 10% lift in PR velocity.
  “statistically significant data that shows somewhere around a 10% lift in PR velocity for Copilot users.”transcript, stated
• The Java-to-Kotlin migration would take about eight years on current trends and about three years with deterministic automation before LLM acceleration.
  “It would take about eight years on the current trends, and we think that's too long.”transcript, stated

  “If we follow this approach, we estimate it'll take us about three years to finish.”transcript, stated

Alternatives considered (roads not taken)

Compete with Copilot using an in-house coding assistant

Rejected because: The project required broad IDE, hosting, fine-tuning, and service investment while vendor products kept improving faster.

Ask ChatGPT to migrate the whole monorepo

Rejected because: A flexible direct LLM migration would be fast, but hallucinations were too risky for mobile production code.

Have platform engineers manually write all AST postprocessor rules

Rejected because: The team expected many reusable rules and footguns, so LLM-assisted rule drafting could accelerate deterministic tooling.

Patterns

tool-use-orchestration (orchestration)

Uber models developer workflows as state machines where deterministic tools run builds, tests, AST transforms, and CI while LLM steps generate or repair artifacts.

human-in-the-loop-gate (governance)

Developers iterate on generated tests, and platform engineers review LLM-authored AST rules before deterministic migration tooling lands.

eval-driven-development (evaluation)

Uber added test validation, mutation-testing ideas, CI checks, and qualitative/quantitative developer productivity metrics to keep AI tooling tied to actual impact.

Tradeoffs

Build on Copilot rather than continue competing internally.

Chose Vendor ecosystem integration over A fully in-house IDE coding assistant.

Rationale: The internal assistant could not keep up with vendor investment and had high integration and maintenance cost.

Use deterministic AST tooling with LLM-authored rules for migration.

Chose LLM-assisted rule generation plus deterministic conversion over Direct LLM migration of source code.

Rationale: This preserved deterministic output and reviewability while using LLMs to speed up rule authoring.

Keep developers in the loop for generated unit tests.

Chose Interactive AutoCover test generation and validation over Automatically checking in generated tests.

Rationale: The team expected an error rate and wanted developers to iterate on generated tests before adoption.

Gotchas (what bit them)

AI MVPs were much easier than production developer tools.

Root cause: Production required IDE integrations, hosting, fine-tuning, service infrastructure, latency, analytics, and maintenance.

Resolution: Pause the in-house assistant and build on the vendor ecosystem where it already had distribution and investment.

Internal assistants risked cannibalizing existing developer UI surfaces.

Root cause: Developers were already using Copilot chats and autocomplete surfaces.

Resolution: Use Copilot extension points and ecosystem integration rather than adding competing chats or hijacking autocomplete.

Direct LLM code migration was too risky for mobile production.

Root cause: Hallucinated code could slip through review and cause outages in conservatively rolled out mobile apps.

Resolution: Use LLMs to write deterministic AST rules, then run conversion, tests, lint, and platform review before automation.

Developer productivity measurement can create bad incentives.

Root cause: Metrics such as lines of code or test coverage can produce verbosity or flaky tests if optimized blindly.

Resolution: Lead with qualitative developer feedback and normalize quantitative impact around developer time saved.

Evaluation

Uber used acceptance-rate and latency goals for the paused assistant, developer skepticism and test validation for AutoCover, CI/build/test/lint loops for migration rules, Copilot adoption/PR velocity for rollout impact, and qualitative developer feedback plus developer-time-saved as broader metrics.

Metrics: 10% target acceptance-rate improvement · less than one second for 100 tokens · around 60% active 30-day Copilot developers · around 10% lift in PR velocity · CI, tests, builds, and lint · developer time saved

Outcomes

• Uber paused the in-house coding assistant and leaned into GitHub Copilot as the platform surface.
• AutoCover advanced into an iterative internal tool for generating and validating tests.
• The Java-to-Kotlin strategy expects to cut migration time in half by combining deterministic tooling with LLM-assisted rule generation.

The work spans six monorepos, over 100 million lines of code, an Android monorepo with over 10 million lines and close to 100,000 files, and about 60% active 30-day Copilot developer usage.