Domain-Adapted LLMs Through Continued Pretraining on E-commerce Data

ebay · E-commerce · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce domain-adapted LLMs

eBay needed domain-specific LLMs for marketplace-scale e-commerce tasks while avoiding the cost, security, and customization limits of third-party frontier models.

Constraints:

• eBay operates with billions of active listings, millions of sellers, and 190 global markets.
• GPT-4 and Claude-style services are costly for high-volume e-commerce workloads.
• Third-party model use can expose proprietary data and limit fine-tuning.
• Training from scratch is time- and resource-intensive.
• Continued pretraining must add domain knowledge without catastrophic forgetting.

Architecture

eBay continued-pretrained Meta Llama 3.1 8B and 70B models on e-commerce and general-domain data to create e-Llama. The training mixture used replay/general examples and 10% non-English general data to preserve base capability, while Megatron-LM distributed training ran across 60 nodes and 480 H100 GPUs. The models were trained on 1 trillion tokens and evaluated against e-commerce and general-domain benchmarks.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  ebay-e-commerce-data["eBay e-commerce data"] --> balanced-training-mixture["Balanced training mixture"]
  general-domain-replay-data["General-domain replay data"] --> balanced-training-mixture["Balanced training mixture"]
  balanced-training-mixture --> llama-3-1-base-models["Llama 3.1 base models"]
  llama-3-1-base-models --> megatron-lm-training-framework["Megatron-LM training framework"]
  h100-compute-platform["H100 compute platform"] --> megatron-lm-training-framework["Megatron-LM training framework"]
  megatron-lm-training-framework --> e-llama-models["e-Llama models"]
  e-llama-models --> benchmark-evaluation["Benchmark evaluation"]

Components

Data flow

eBay marketplace data and curated general-domain data are combined into a balanced training mixture. Llama 3.1 base models undergo continued pretraining with Megatron-LM on the eBay compute platform. e-Llama models are then evaluated for e-commerce gains, non-English gains, and general-domain degradation before powering company AI initiatives.

Why this architecture

Continued pretraining let eBay reuse open Llama models rather than train every model from scratch, while keeping more control and lower cost than third-party APIs. Replay/general data balanced the domain gains against catastrophic forgetting.

Models used: Llama 3.1 8B · Llama 3.1 70B · e-Llama · LiLiuM · Megatron-LM · GPT-4 · Claude

Scale

• eBay has billions of active listings, millions of sellers, and 190 global markets.
  “there are billions of active listings on our platform at any moment with millions of active sellers in 190 global markets.”article, stated
• The adapted models are 8B and 70B parameter e-Llama models.
  “8-billion and 70-billion parameter language models which we have adapted to the e-commerce domain. In short, “e-Llama.””article, stated
• Training used 60 nodes with 8 NVIDIA H100 GPUs each, totaling 480 GPUs.
  “Training was conducted using 60 nodes, each having 8 NVIDIA H100 80GB GPUs (a total of 480 GPUs).”article, stated
• Training used 1 trillion tokens; the 70B model took around one month / 340k GPU-hours.
  “Training the 70 billion model on 1 trillion tokens took around one month, or about 340k GPU-hours.”article, stated

Alternatives considered (roads not taken)

Use GPT-4 or Claude services for scaled e-commerce tasks

Rejected because: They are costly, less controllable, and create proprietary-data/security concerns.

Train every model from scratch

Rejected because: The article calls scratch training time- and resource-intensive.

Use only domain/e-commerce data

Rejected because: The model would risk forgetting general pretrained knowledge.

Patterns

fine-tuning-adaptation (model-ops)

eBay used continued pretraining to adapt Llama 3.1 base models to e-commerce.

data-quality-loop (data)

The training mixture includes replay/general data and non-English data to retain general and multilingual behavior.

cost-aware-routing (strategy)

The domain-adapted model path reduces dependence on costly third-party services at eBay marketplace scale.

privacy-preserving-processing (security)

In-house model capability reduces the need to send proprietary marketplace information to third-party models.

Tradeoffs

Adopt a hybrid LLM strategy

Chose Scratch-built LiLiuM plus adapted Llama/e-Llama over Only third-party models or only scratch-built models.

Rationale: The hybrid approach balances control, speed, cost, and domain customization.

Continue-pretrain open models

Chose Llama 3.1 domain adaptation over Training from scratch for this model family.

Rationale: Using existing pretrained models moves faster while domain data infuses e-commerce knowledge.

Preserve general-domain ability

Chose Replay/general and 10% non-English data over All e-commerce-domain data.

Rationale: The model needed e-commerce gains without excessive degradation on general tasks.

Gotchas (what bit them)

Continued pretraining can erase general capability.

Root cause: Overweighting domain data can cause catastrophic forgetting.

Resolution: Use replay/general-domain data and balance the data ratio.

Third-party LLMs become impractical at e-commerce scale.

Root cause: Cost, proprietary-data exposure, and limited fine-tuning all compound at high volume.

Resolution: Build a hybrid in-house/open-model strategy.

Large-scale training needs distributed optimization.

Root cause: States must be distributed across GPUs/nodes for 70B-scale continued pretraining.

Resolution: Use Megatron-LM 3D parallelism, distributed optimizer states, and flash-attention optimizations.

Evaluation

eBay evaluated e-Llama on English/non-English e-commerce-specific benchmarks and general-domain NLU benchmarks to measure both improvement and forgetting.

Metrics: 25% English e-commerce benchmark improvement · 30% non-English e-commerce benchmark improvement · 1% degradation on general-domain NLU benchmarks · Training efficiency: 1 trillion tokens, about one month, about 340k GPU-hours

Outcomes

• Created e-Llama 8B and 70B domain-adapted models.
• Improved e-commerce benchmarks by about 25% in English and 30% in non-English.
• Kept general-domain NLU degradation to about 1% for e-Llama 70B.
• Enabled eBay to drive new AI initiatives with proprietary and open LLMs.

This duplicate eBay source describes marketplace-scale domain adaptation with billions of listings, millions of sellers, 190 markets, 480 H100 GPUs, and 1 trillion training tokens.