Multi-Track Approach to Developer Productivity Using LLMs

ebay · E-commerce · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce

eBay wanted to improve developer productivity across several use cases rather than relying on one generic LLM tool.

Constraints:

Architecture

eBay treated developer productivity as a portfolio of complementary AI tracks rather than a single tool rollout. The commercial track expanded GitHub Copilot and measured it with a 300-developer A/B pilot plus surveys and delivery metrics. The custom-model track created eBayCoder by fine-tuning Code Llama 13B on eBay code and documentation so upkeep tasks could use broader internal context than a commercial assistant can fit. The knowledge track built an internal RAG GPT that ingests enterprise GitHub Markdown, Google Docs, Jira, Slack, and Wikis, embeds content into a vector database, retrieves similar context for a developer query, and calls private commercial/open-source LLMs with instructions to answer only from the provided context.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  copilot-pilot["copilot-pilot"] --> developer-telemetry["developer-telemetry"]
  copilot-pilot --> developer-surveys["developer-surveys"]
  internal-code-and-docs["internal-code-and-docs"] --> ebaycoder["ebaycoder"]
  internal-code-and-docs --> embedding-pipeline["embedding-pipeline"]
  embedding-pipeline --> vector-database["vector-database"]
  vector-database --> internal-gpt["internal-gpt"]
  internal-gpt --> private-llm-instance["private-llm-instance"]
  internal-gpt --> feedback-loop["feedback-loop"]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
copilot-pilot Commercial coding-assistant track measured with a pilot/control-group A/B experiment and developer surveys. developer-telemetry, developer-surveys
developer-telemetry Collects quantitative signals such as code acceptance, PR creation-to-merge time, lead time for change, and code quality. copilot-pilot
developer-surveys Collects qualitative perceived-productivity feedback to complement delivery metrics. copilot-pilot
ebaycoder Fine-tuned Code Llama 13B model trained on eBay codebase and documentation for company-specific code upkeep tasks. internal-code-and-docs
internal-code-and-docs Internal codebase and associated documentation used for post-training/fine-tuning and for RAG ingestion. ebaycoder, embedding-pipeline
embedding-pipeline Creates vector embeddings for internal knowledge content. internal-code-and-docs, vector-database
vector-database Stores content embeddings and retrieves the most similar internal context for a developer query. embedding-pipeline, internal-gpt
internal-gpt Private GPT interface that answers developer questions using retrieved internal context. vector-database, private-llm-instance, feedback-loop
private-llm-instance Commercial or open-source LLM called with retrieved context and an instruction to answer only when the context contains the answer. internal-gpt
feedback-loop Collects employee feedback from the internal GPT UI to improve answers over time. internal-gpt

Data flow

A developer-productivity need is routed to one of three tracks. General coding assistance goes through Copilot and is evaluated with pilot/control-group telemetry and surveys. Codebase-wide upkeep tasks use eBayCoder, a Code Llama 13B model trained on eBay's codebase and documentation. Internal knowledge questions enter the internal GPT: source documents are ingested from GitHub, Docs, Jira, Slack, and Wikis, converted to embeddings, stored in a vector database, retrieved by semantic similarity against the developer query, and passed with context to private LLM instances that must say "I don't know" when the answer is absent.

4 supporting quotes
“we believe we’ve unlocked three major tracks to developer productivity: utilizing a commercial offering, fine-tuning an existing Large Language Model (LLM), and leveraging an internal network.”article, stated
“we created what we call eBayCoder: Code Llama that’s trained on eBay’s codebase and associated documentation.”article, stated
“a system that creates an embedding vector for each piece of content, which is then stored in a vector database.”article, stated
“When someone enters a query, the system creates an embedding vector, and then uses a similarity mechanism (such as cosine similarity) to compare the query vector against all the known aforementioned content embedding vectors.”article, stated

Why this architecture

The source explicitly rejects a single-best-track framing. Copilot works well for broad coding assistance but hits context limits at eBay scale. eBayCoder addresses tasks that need internal codebase knowledge. The internal GPT addresses distributed knowledge and meeting/document lookup. Evaluation combines quantitative delivery signals with qualitative surveys because developer productivity has no reliable single metric.

3 supporting quotes
“it’s not a matter of ranking them “good, better, or best.” Each can be used separately or in any combination, and bring their own benefits and drawbacks.”article, stated
“there is no reliable single metric to measure developer productivity given its complexity.”article, stated
“For a company of eBay’s size, though, certain tasks are simply not possible. We’ve got millions of lines of code, and some activities require the knowledge of the entire eBay codebase.”article, stated

Models used: GitHub Copilot · Code Llama 13B

Scale

Alternatives considered (roads not taken)

Picking one AI developer-productivity track as the universal answer

Rejected because: The tracks have different benefits and drawbacks, and eBay found they can be used separately or in combination.

“it’s not a matter of ranking them “good, better, or best.” Each can be used separately or in any combination, and bring their own benefits and drawbacks.”article, stated

Relying only on commercial Copilot for all developer tasks

Rejected because: Copilot's prompt/context limits make whole-codebase activities impossible at eBay scale.

“Copilot has a limit to its prompt size – essentially, how much data it can process.”article, stated
“For a company of eBay’s size, though, certain tasks are simply not possible. We’ve got millions of lines of code, and some activities require the knowledge of the entire eBay codebase.”article, stated

Leaving internal documentation lookup to meetings, dead ends, and informal experts

Rejected because: The internal GPT centralizes access to distributed primary sources and lets experienced developers focus on innovation.

“Trying to find the answers to seemingly simple questions can sometimes require multiple meetings, dead ends, and red herrings – all of which adds up to reduced productivity and increased annoyance.”article, stated
“We want our most knowledgeable and experienced developers to focus on innovation.”article, stated

Patterns

eval-driven-development (evaluation)

eBay evaluated Copilot with a pilot/control-group A/B test and developer surveys rather than relying on anecdotes alone.

“Our test for this track involved a, well, pilot for Copilot, comprising 300 developers – half using Copilot, and half forming a control group with similar assignments and abilities but without Copilot, in an A/B test experiment conducted in the summer of 2023.”article, stated

fine-tuning-adaptation (model-operations)

eBay created a fine-tuned LLM path for software upkeep tasks that needed internal codebase context.

“But a fine-tuned LLM can access as much context as we want, potentially reducing the amount of code duplication.”article, stated

rag-retrieval (retrieval)

The third track used RAG to reduce time spent hunting through internal documentation and meetings.

“Trying to find the answers to seemingly simple questions can sometimes require multiple meetings, dead ends, and red herrings – all of which adds up to reduced productivity and increased annoyance.”article, stated

Tradeoffs

Treat developer productivity AI as a portfolio instead of selecting a single winner.

Chose three complementary tracks: Copilot, eBayCoder, and an internal RAG GPT over ranking one track as universally best.

Rationale: The source says each track can be used separately or together and carries different benefits and drawbacks.

“it’s not a matter of ranking them “good, better, or best.” Each can be used separately or in any combination, and bring their own benefits and drawbacks.”article, stated

Train a company-specific code model for context-heavy upkeep tasks.

Chose eBayCoder trained on eBay codebase and documentation over commercial assistants limited to local prompt context.

Rationale: Internal codebase context can reduce duplicate work and support upkeep tasks that require knowledge beyond surrounding files.

“a fine-tuned LLM can access as much context as we want, potentially reducing the amount of code duplication.”article, stated
“With the existing migration tools at eBay, we still spend significant engineering resources on software upkeep.”article, stated

Gotchas (what bit them)

Commercial coding assistants were not enough for tasks requiring broad internal context.

Root cause: Copilot could not process unlimited context from eBay's codebase.

Resolution: eBay added additional tracks, including a fine-tuned model and RAG knowledge base.

“Copilot has a limit to its prompt size – essentially, how much data it can process.”article, stated

Evaluation

eBay combined quantitative tools and qualitative surveys, including an A/B Copilot pilot.

Metrics: 27% code acceptance rate · developer surveys

“Following a two-week ramp-up period, developer surveys showed that the developers using Copilot saw an increase in perceived productivity, along with a 27% code acceptance rate (as reported through Copilot telemetry).”article, stated

Outcomes

The article describes expansion of Copilot to all developers.

“We expanded our use of GitHub Copilot, a commercial offering, to all of our developers last year.”article, stated