Multi-Track Approach to Developer Productivity Using LLMs

ebay · E-commerce · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Developer productivity at enterprise scale

eBay needed to improve developer productivity with generative AI while recognizing that no single tool, model, or metric could cover code completion, internal migrations, duplication reduction, and knowledge discovery across a large engineering organization.

Constraints:

Developer productivity is complex and cannot be measured by one metric.
Commercial coding assistants have prompt-size and context limitations.
Fine-tuned models need internal codebase and documentation context for enterprise-specific work.
Internal information is distributed across GitHub, Google Docs, Jira, Slack, and wikis.
Each productivity track has different integration resources, benefits, and drawbacks.

Architecture

eBay pursued three complementary tracks. Track 1 expanded GitHub Copilot after a 300-developer A/B test. Track 2 fine-tuned Code Llama 13B on eBay's codebase and documentation to create eBayCoder for internal code migration, duplication reduction, and enterprise-specific code tasks. Track 3 built an internal RAG-based GPT that ingests selected primary sources, embeds content into a vector database, retrieves relevant internal knowledge for developer questions, and reduces meeting-heavy discovery work.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  developer-productivity-program[Developer productivity program] --> github-copilot-track[GitHub Copilot track]
  developer-productivity-program --> ebaycoder-track[eBayCoder track]
  developer-productivity-program --> internal-gpt-track[Internal GPT track]
  internal-codebase-and-documentation[Internal codebase and documentation] --> ebaycoder-track
  internal-codebase-and-documentation --> internal-gpt-track
  internal-gpt-track --> vector-database[Vector database]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Developer productivity program`	Evaluation and adoption program spanning commercial tools, fine-tuned models, and internal knowledge retrieval.	`GitHub Copilot track`, `eBayCoder track`, `Internal GPT track`
`GitHub Copilot track`	Commercial coding assistant evaluated with A/B testing and developer telemetry.	`Developer productivity program`
`eBayCoder track`	Code Llama 13B fine-tuned on eBay's codebase and documentation.	`Internal codebase and documentation`
`Internal codebase and documentation`	Organization-specific source data for fine-tuning and retrieval.	`eBayCoder track`, `Internal GPT track`
`Internal GPT track`	RAG system over selected primary internal sources.	`Vector database`
`Vector database`	Stores embeddings for internal source content and supports developer query retrieval.	`Internal GPT track`

Data flow

Developers use different AI paths depending on the task: Copilot supports local coding assistance, eBayCoder uses internal code/documentation fine-tuning for enterprise code tasks, and the internal GPT embeds internal source content into a vector database so developer queries retrieve relevant documents and generate answers.

“we believe we’ve unlocked three major tracks to developer productivity: utilizing a commercial offering, fine-tuning an existing Large Language Model (LLM), and leveraging an internal network.”article, stated

“we created what we call eBayCoder: Code Llama that’s trained on eBay’s codebase and associated documentation.”article, stated

Why this architecture

eBay treats the tracks as complementary because commercial code completion, internal-code fine-tuning, and organizational knowledge retrieval solve different productivity bottlenecks. The measurement layer also combines quantitative telemetry and qualitative surveys because developer productivity is not reducible to one metric.

“it’s not a matter of ranking them “good, better, or best.” Each can be used separately or in any combination, and bring their own benefits and drawbacks.”article, stated

“there is no reliable single metric to measure developer productivity given its complexity.”article, stated

Models used: GitHub Copilot · Code Llama 13B · eBayCoder · internal GPT · RAG

Scale

The Copilot pilot involved 300 developers.
“Our test for this track involved a, well, pilot for Copilot, comprising 300 developers – half using Copilot, and half forming a control group”article, stated
Copilot telemetry showed a 27% code acceptance rate.
“the developers using Copilot saw an increase in perceived productivity, along with a 27% code acceptance rate”article, stated
The Copilot group saw a 17% PR creation-to-merge time decrease.
“for the experimental Copilot group, we saw a 17% decrease in pull request (PR) creation to merge time”article, stated
The Copilot group saw a 12% Lead Time for Change decrease.
“and a 12% decrease in [Lead Time for Change]”article, stated
eBayCoder used Code Llama 13B as the base model.
“We used Code Llama 13B as our base LLM for this exercise”article, stated

Alternatives considered (roads not taken)

Pick one AI tool as the best answer.

Rejected because: eBay argues the three tracks have different benefits/drawbacks and can be combined.

“Each can be used separately or in any combination, and bring their own benefits and drawbacks.”article, stated

Rely only on commercial assistants such as Copilot.

Rejected because: Copilot has prompt-size/context limits and cannot incorporate all internal enterprise code context.

“Copilot has a limit to its prompt size – essentially, how much data it can process.”article, stated

Let developers find internal information manually.

Rejected because: Simple questions could require multiple meetings, dead ends, and red herrings.

“Trying to find the answers to seemingly simple questions can sometimes require multiple meetings, dead ends, and red herrings”article, stated

Patterns

`fine-tuning-adaptation` (model-operations)

eBay fine-tuned Code Llama 13B on its internal codebase and associated documentation to create eBayCoder.

“we created what we call eBayCoder: Code Llama that’s trained on eBay’s codebase and associated documentation.”article, stated

`rag-retrieval` (retrieval)

The internal GPT uses RAG with embeddings in a vector database over internal primary sources.

“we used an Retrieval Augmented Generation (RAG) – a system that creates an embedding vector for each piece of content”article, stated

`eval-driven-development` (evaluation)

eBay measured Copilot with a controlled A/B test and used multiple quantitative and qualitative metrics.

“half using Copilot, and half forming a control group with similar assignments and abilities but without Copilot, in an A/B test experiment”article, stated

`context-compaction` (context)

The case highlights context-window limits in commercial assistants and uses internal fine-tuning/RAG for broader enterprise context.

“For a company of eBay’s size, though, certain tasks are simply not possible. We’ve got millions of lines of code”article, stated

Tradeoffs

Adoption strategy

Chose Three complementary AI productivity tracks. over Ranking one approach as globally best..

Rationale: Each track serves different workflows and can be combined despite different integration costs.

“Each of these tracks requires additional resources to integrate, but it’s not a matter of ranking them “good, better, or best.””article, stated

Internal model customization

Chose Fine-tune open-source Code Llama on eBay data. over Using only a model without internal data..

Rationale: Internal data enables enterprise-specific migration and duplication-reduction tasks.

“there’s only so much we can learn from a model that doesn’t incorporate our internal data.”article, stated

Knowledge assistant architecture

Chose Automated recurring RAG ingestion from primary sources. over Meeting-driven manual discovery..

Rationale: Developers need quick answers from distributed internal information.

“So we created an internal GPT that ingests certain data from relevant primary sources.”article, stated

Gotchas (what bit them)

Developer productivity cannot be captured by a single metric.

Root cause: Productivity includes code, flow, quality, performance, and perceived developer experience.

Resolution: eBay combined Git/DORA/Flow/code-quality metrics with targeted developer surveys.

“Instead, as suggested by studies, we use both quantitative and qualitative metrics.”article, stated

Commercial assistants lack enterprise-wide context.

Root cause: Copilot sees only immediately relevant repository context and a few dependent libraries.

Resolution: eBay trained eBayCoder on the internal codebase and documentation.

“It may not be aware of a different internal service or a non-dependent library maintained by other groups that offer the same functionality currently being authored.”article, stated

Internal knowledge discovery wastes developer time.

Root cause: Information is scattered across many enterprise systems and social channels.

Resolution: eBay built an internal GPT over relevant primary sources.

“Internal information is distributed across primary sources including enterprise GitHub Markdowns, Google Docs, Jira, Slack, and Wikis.”article, stated

Evaluation

eBay evaluated Copilot through an A/B test with 300 developers, telemetry, developer surveys, code accuracy, document accuracy, PR merge time, Lead Time for Change, and Sonar code quality.

Metrics: 300 developers in Copilot pilot · 27% code acceptance rate · 70% generated documentation accuracy · 60% generated code accuracy · 17% decrease in PR creation-to-merge time · 12% decrease in Lead Time for Change · Sonar code quality remained the same

“the generated documents were 70% accurate, and the generated code was at 60%.”article, stated

“The code quality measure through Sonar remained the same for both groups.”article, stated

Outcomes

Copilot users saw perceived productivity gains and measurable reductions in PR and lead-time metrics.
eBayCoder helped make labor- and time-intensive internal code tasks easier.
Internal GPT reduced dependency on meetings and long-tenured employees for organizational knowledge.
eBay established a multi-track AI productivity strategy rather than one universal tool choice.

The Copilot A/B test involved 300 developers; eBayCoder was built from Code Llama 13B; the internal GPT ingests primary sources such as GitHub Markdown, Google Docs, Jira, Slack, and wikis.

“We found that eBayCoder was able to make some tasks that were previously labor- and time-intensive much easier.”article, stated

“We want our most knowledgeable and experienced developers to focus on innovation.”article, stated