Developing and Deploying Domain-Adapted LLMs for E-commerce Through Continued Pre-training

ebay · E-commerce · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce domain-adapted LLMs

eBay needed LLMs with e-commerce domain knowledge that could scale across a global marketplace without the cost, data-security, and customization limits of third-party proprietary models.

Constraints:

The platform has billions of active listings, millions of active sellers, and 190 global markets.
Third-party models such as GPT-4 and Claude are too costly for high-volume e-commerce use cases.
Sending proprietary data to external services introduces data-security risk.
Training a large LLM from scratch is time- and resource-intensive.
Domain adaptation must improve e-commerce capability without catastrophic forgetting of general-language skills.

Architecture

eBay used a hybrid LLM strategy: it builds models from scratch such as LiLiuM for full control, while also adapting open Llama 3.1 models through continued pretraining on mixed eBay e-commerce and general-domain data. The e-Llama work produced 8B and 70B parameter models trained on 1 trillion tokens using Megatron-LM 3D parallelism across 480 NVIDIA H100 GPUs, replay-style general data, and a balanced 1:1 general-to-e-commerce data ratio.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  ebay-marketplace-data["eBay marketplace data"] --> training-data-mixture["Training data mixture"]
  general-domain-replay-data["General-domain replay data"] --> training-data-mixture["Training data mixture"]
  training-data-mixture --> llama-3-1-base-models["Llama 3.1 base models"]
  llama-3-1-base-models --> megatron-lm-training-stack["Megatron-LM training stack"]
  ebay-h100-compute-platform["eBay H100 compute platform"] --> megatron-lm-training-stack["Megatron-LM training stack"]
  megatron-lm-training-stack --> e-llama-models["e-Llama models"]
  e-llama-models --> benchmark-evaluation["Benchmark evaluation"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`eBay marketplace data`	Proprietary e-commerce corpus and decades of domain insights.	`Training data mixture`
`General-domain replay data`	Curated/public/general and non-English examples used to reduce catastrophic forgetting.	`Training data mixture`
`Training data mixture`	Balanced general-to-e-commerce continued-pretraining corpus.	`Llama 3.1 base models`
`Llama 3.1 base models`	8B and 70B pretrained model starting points.	`Megatron-LM training stack`
`Megatron-LM training stack`	Distributed training framework with 3D parallelism, optimizer sharding, and flash-attention optimizations.	`eBay H100 compute platform`, `e-Llama models`
`eBay H100 compute platform`	60-node, 480-GPU training hardware.	`Megatron-LM training stack`
`e-Llama models`	Domain-adapted 8B/70B Llama models for e-commerce tasks.	`Benchmark evaluation`
`Benchmark evaluation`	E-commerce and general-domain benchmarks measuring improvement and forgetting.	`e-Llama models`

Data flow

eBay combines e-commerce data with curated general-domain and non-English data. Llama 3.1 base models are continued-pretrained with replay data to retain general knowledge, using Megatron-LM distributed training over the eBay compute platform. The resulting e-Llama models are evaluated on e-commerce-specific and general-domain benchmarks before being used across company AI initiatives.

“we continue training the Llama base models on a large amount of e-commerce data in order to infuse domain specific knowledge into the model.”article, stated

“We use Megatron-LM as a highly-optimized training framework that allows us to use 3D parallelism in training”article, stated

Why this architecture

Continued pretraining gave eBay a faster path than training every model from scratch while avoiding the cost, data-security, and fine-tuning limits of third-party proprietary models. The data mixture and replay strategy were designed to add e-commerce knowledge without degrading general-domain behavior.

“This allows us to move faster and unlock more value, as we do not have to develop the models completely from scratch.”article, stated

“the training setup has to be carefully balanced to make the model learn about the new domain while preventing the model from degrading too much in performance on general domain tasks.”article, stated

Models used: Llama 3.1 8B · Llama 3.1 70B · e-Llama · LiLiuM · Megatron-LM · GPT-4 · Claude

Scale

eBay has billions of active listings, millions of active sellers, and 190 global markets.
“there are billions of active listings on our platform at any moment with millions of active sellers in 190 global markets.”article, stated
The adapted models include 8-billion and 70-billion parameter e-Llama models.
“8-billion and 70-billion parameter language models which we have adapted to the e-commerce domain. In short, “e-Llama.””article, stated
Training used 60 nodes with 8 NVIDIA H100 80GB GPUs each, for 480 GPUs total.
“Training was conducted using 60 nodes, each having 8 NVIDIA H100 80GB GPUs (a total of 480 GPUs).”article, stated
Training used a batch size of about 11.8 million tokens and 85k update steps.
“We use cosine learning rate scheduling with warmup, a batch size of approximately 11.8 million tokens and 85k total update steps.”article, stated
The models were trained on 1 trillion tokens; the 70B run took about one month or 340k GPU-hours.
“the models are trained on 1 trillion tokens in total. Training the 70 billion model on 1 trillion tokens took around one month, or about 340k GPU-hours.”article, stated

Alternatives considered (roads not taken)

Rely on GPT-4 or Claude for scaled e-commerce workloads

Rejected because: Third-party models were costly, carried data-security risks, and limited fine-tuning on proprietary data.

“these services come with considerable costs, making them impractical for businesses like eBay that need fine-tuned, scalable and cost-effective solutions.”article, stated

Train every e-commerce model from scratch

Rejected because: Training from scratch is time/resource intensive, while adapting Llama lets eBay move faster.

“Training a large-scale LLM from scratch is a very time- and resource-intensive process.”article, stated

Train only on e-commerce data

Rejected because: A pure domain corpus risks catastrophic forgetting of general-domain behavior.

“The goal is to infuse e-commerce specific knowledge into the Llama 3.1 base models, without the models forgetting information they have learned in their original pretraining”article, stated

Patterns

`fine-tuning-adaptation` (model-ops)

eBay continued-pretrained Llama 3.1 models on e-commerce and general data to create e-Llama.

“This technique is known as "continued pretraining"”article, stated

`data-quality-loop` (data)

The data mixture balances e-commerce examples with replay/general-domain and non-English data to improve domain performance while preserving general capability.

“We also include 10% non-English language general domain data to further enhance the model’s multilingual capabilities.”article, stated

`cost-aware-routing` (strategy)

The in-house/domain-adapted model strategy reduces dependence on costly third-party models for high-volume marketplace tasks.

“these services come with considerable costs, making them impractical for businesses like eBay”article, stated

`privacy-preserving-processing` (security)

The in-house approach reduces exposure of proprietary marketplace data to third-party model services.

“relying on third-party models introduces data security risks and limits fine-tuning capabilities based on proprietary data.”article, stated

Tradeoffs

Use a hybrid model strategy

Chose Scratch-built LiLiuM plus adapted open Llama models over Only building from scratch or only consuming third-party models.

Rationale: The hybrid path balances control with speed and reuse.

“because of the depth and breadth of our data, we have taken a hybrid approach to LLMs.”article, stated

Continue-pretrain Llama 3.1

Chose Adapt existing pretrained models over Training every e-commerce LLM from scratch.

Rationale: Existing pretrained models move faster, but need e-commerce knowledge injected.

“In order to move fast, one could use existing pretrained models, such as Llama 3.1, for their use cases.”article, stated

Balance domain and general data

Chose 1:1 general-to-e-commerce sampling plus replay over Maximizing e-commerce data at the expense of general-domain retention.

Rationale: The team wanted domain gains while avoiding general-task degradation.

“a maximum learning rate of 10% of the original maximum learning rate, and a general-to-e-commerce data sampling ratio of 1-1 gives the best results.”article, stated

Gotchas (what bit them)

Domain adaptation can cause catastrophic forgetting.

Root cause: Training too heavily on e-commerce data can degrade general-domain performance.

Resolution: Use replay/general-domain examples and tune the general-to-e-commerce data ratio.

“This "replay" has been shown to help a model retain previously learned information.”article, stated

Third-party frontier models are not automatically viable at marketplace scale.

Root cause: Cost, data security, and limited proprietary fine-tuning make them impractical for some high-volume e-commerce needs.

Resolution: Combine proprietary/in-house models with adapted open models.

“To address these challenges, we aimed to build an in-house solution.”article, stated

Large-scale LLM training needs serious distributed-training engineering.

Root cause: Model and optimizer states have to be distributed over many GPUs and nodes.

Resolution: Use Megatron-LM 3D parallelism, distributed optimizer states, and optimized attention.

“Model training at this scale requires an efficient distribution of model and optimizer states across several GPUs and sometimes even across nodes.”article, stated

Evaluation

eBay evaluated e-Llama on e-commerce-specific benchmarks in English and non-English, plus general-domain NLU benchmarks to measure catastrophic forgetting.

Metrics: Approximately 25% English e-commerce benchmark improvement · Approximately 30% non-English e-commerce benchmark improvement · Only 1% degradation on general-domain NLU benchmarks for e-Llama 70B · Training efficiency: 1 trillion tokens in about one month / 340k GPU-hours

“approximately 25% improvement in e-commerce-specific benchmarks for English and about 30% improvement for non-English”article, stated

“we observe only 1% degradation on general domain NLU benchmarks for the large e-Llama 70B model.”article, stated

Outcomes

Created e-Llama 8B and 70B domain-adapted models for e-commerce.
Improved e-commerce-specific benchmark performance by about 25% in English and 30% in non-English.
Limited general-domain NLU degradation to 1% for the 70B model.
Enabled eBay to leverage proprietary and open LLMs for company-wide AI initiatives.

The work targets eBay's marketplace scale: billions of listings, millions of sellers, 190 global markets, 480 H100 GPUs, and 1 trillion training tokens.

3 supporting quotes

“the model training is enabling eBay to leverage proprietary and open LLMs to drive new AI initiatives across the company.”article, stated

“The final e-Llama models demonstrate approximately 25% improvement in e-commerce-specific benchmarks for English and about 30% improvement for non-English”article, stated

“Training the 70 billion model on 1 trillion tokens took around one month, or about 340k GPU-hours.”article, stated