Large-Scale LLM Infrastructure for E-commerce Applications

coupang · E-commerce · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce LLM platform infrastructure

Coupang needed to support LLM and multimodal model use across search, ads, catalog, recommendations, operations, and multilingual product understanding. The platform had to work in Korean, Mandarin, and English; handle limited labels; support fine-tuning, RAG, weak-label generation, categorization, and embeddings; and scale training and inference despite GPU shortages, high GPU cost, CPU-only legacy serving, and model-parallel training requirements.

Constraints:

• Coupang operates shopping, eats, and play products with large product, user, and merchant datasets
• Korean and Mandarin training data are relatively small for many tasks
• large language models are compute and memory intensive for both training and serving
• cloud high-RAM GPUs such as A100 and H100 have significant wait times
• the existing training stack was not equipped for model-parallel LLM training
• pre-LLM model serving was CPU-only and too slow for multi-billion-parameter models
• real-time e-commerce serving needs different latency and resource tradeoffs than batch or nearline workflows

Architecture

Coupang built a pragmatic LLM platform on Kubernetes using open-source model architectures, in-context learning, supervised fine-tuning, continued pretraining, weak-label generation, distillation, embedding extraction, batch inference, nearline inference, and GPU-backed serving. Exploration uses Zeppelin, Spark on Kubernetes, and GPU Jupyter notebooks. Training uses Polyaxon, PyTorchJob, model parallel training, DeepSpeed ZeRO, Hugging Face recipes, and shared hyperparameter defaults. Production paths use distilled smaller models for real-time inference, exported embeddings for ranking, Ray plus vLLM for batch and nearline pipelines, and Nvidia Triton on AWS EKS for real-time GPU serving. Infrastructure spans multi-region cloud clusters in APAC and the US plus on-prem high-end GPU clusters to mitigate GPU shortages and reduce training cost.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  teams["Product ML teams"] --> explore["Exploration notebooks"]
  explore --> training["Training platform"]
  training --> deepspeed["DeepSpeed ZeRO recipes"]
  training --> gpu["Hybrid GPU clusters"]
  training --> source["Source LLMs"]
  source --> distill["Distillation pipeline"]
  source --> embeddings["Embedding export pipeline"]
  source --> batch["Batch and nearline inference"]
  distill --> triton["Realtime Triton serving"]
  batch --> downstream["Downstream ML systems"]
  embeddings --> downstream
  triton --> inference["Inference platform"]
  batch --> inference
  gpu --> triton
  gpu --> batch
  inference --> teams

Components

Data flow

ML teams explore model lines, sizes, and prompts in notebooks, then shortlist models for fine-tuning or continued pretraining based on compute budget, dataset size, and measured application performance. Source LLMs generate weak labels, produce embeddings, or feed distillation pipelines. For production, real-time workloads run on Triton/EKS, while batch and nearline workflows use Ray plus vLLM to combine CPU preprocessing with GPU inference. Outputs feed downstream recommendation, search, ads, catalog, categorization, attribute extraction, and ranking systems. Benchmarking and experimentation feed back into GPU choice, model size, quantization, offloading recipes, and framework selection.

Why this architecture

Coupang chooses open-source architectures and adaptation rather than training foundation models from scratch because it saves time and compute while still allowing domain-specific performance. It separates exploration, training, batch/nearline inference, distillation, and real-time serving because LLM use cases have different latency, cost, and hardware needs. Hybrid cloud/on-prem multi-region GPU sourcing reduces cloud capacity waits, while tiered GPU selection and smaller model variants prevent high-end GPUs from being wasted on early experiments.

Models used: Qwen · LLAMA · T5 · Phi · Polyglot · CLIP · TrOCR · DeepSpeed ZeRO · vLLM · Nvidia Triton · Ray · Polyaxon · PyTorchJob

Scale

• Coupang uses ML across shopping, eats, and play surfaces.
  “recommendation surfaces such as main home feed, search and ads across Coupang apps for shopping, eats and play.”article, stated
• Coupang has over 100 fulfillment centers housing millions of products.
  “Coupang has over 100+ unique fulfillment centers housing millions of products.”article, stated
• The favored language model sizes are 3B to 20B parameters.
  “Parameter sizes ranging from 3B to 20B are favored”article, stated
• Coupang explicitly compares 70B+ variants with sub-10B variants during exploration.
  “Return on Investment (ROI) of using larger sizes for example 70B+ variant vs < 10B parameter variant”article, stated
• Training models above 1B parameters in mixed precision uses A100-80GB GPUs.
  “For the training of models with more than 1 billion parameters in mixed precision mode, we utilized the A100–80 GB version.”article, stated
• Testing and lightweight training use A10G-24GB devices.
  “For testing and lightweight training purposes, we could employ a substantial quantity of A10G-24 [3.3] GB devices.”article, stated
• vLLM improved throughput by nearly 20x in multiple workloads.
  “vLLM provided us nearly ~20x throughput improvement in multiple workloads”article, stated

Alternatives considered (roads not taken)

Train foundation models from scratch as the default

Rejected because: Coupang favors open-source architectures plus domain fine-tuning to save time and compute.

Serve large source LLMs directly for all real-time use cases

Rejected because: Coupang often distills smaller models or exports embeddings for downstream models to make production serving practical.

Use data parallelism alone for LLM training

Rejected because: Large models do not fit into a single GPU, so Coupang uses model sharding strategies such as DeepSpeed ZeRO.

Use only one inference mode

Rejected because: Coupang built separate real-time, batch, and nearline inference paths because workloads differ in latency, throughput, CPU/GPU mix, and resource footprint.

Patterns

fine-tuning-adaptation (model-adaptation)

Coupang fine-tunes open-source LLM architectures with domain data for business tasks instead of building foundation models from scratch.

model-distillation (serving)

Coupang distills smaller models from trained source LLMs so real-time inference can run with lower resource cost.

batch-offline-processing (inference)

Batch GPU inference extracts source-LLM predictions at scale for distillation and embeddings, using Ray plus vLLM for heterogeneous CPU/GPU pipelines.

cost-aware-routing (infrastructure)

Coupang matches model sizes and GPU types to workload phase, uses multi-region/on-prem capacity, and avoids expensive devices for early experiments.

rag-retrieval (grounding)

Coupang uses RAG where generated responses need grounding in external knowledge such as document corpora or product catalogs.

latency-optimization (serving)

Coupang uses Triton dynamic batching, Ray/vLLM throughput improvements, nearline small batches, and model distillation to fit latency/resource constraints.

data-quality-loop (labeling)

LLM-generated weak labels pass quality checks and then train downstream models, helping with label scarcity in multilingual segments.

Tradeoffs

Primary model adaptation strategy

Chose Open-source architecture plus fine-tuning over Training foundation models from scratch.

Rationale: Fine-tuning lets teams use state-of-the-art pre-trained models with less time and compute.

Prototype pattern

Chose In-context learning before heavier training over Fine-tuning or pretraining first for every idea.

Rationale: ICL is fast, cheap, reusable across tasks, and useful for evaluating whether LLMs can solve a product problem.

Production serving path

Chose Distillation or embedding extraction for many use cases over Serve the full source LLM directly in real time.

Rationale: Smaller real-time models and embeddings bridge LLM quality into production systems with lower serving cost and latency.

GPU sourcing

Chose Hybrid cloud multi-region plus on-prem clusters over Single-region cloud GPU dependence.

Rationale: Multi-region and on-prem capacity mitigated cloud GPU wait times and reduced overall training cost.

Gotchas (what bit them)

Under-resourced languages change both model and tokenizer choices.

Root cause: Korean and Mandarin have limited training data, character-based structure, spacing differences, and larger vocabularies.

Resolution: Favor multilingual-capable architectures and domain adaptation rather than assuming English-centric models are sufficient.

GPU shortages can block execution plans.

Root cause: High-memory GPUs are expensive, scarce, and have significant wait times from cloud vendors.

Resolution: Use multi-region cloud clusters, on-prem high-end GPUs, and lower-performance GPUs for testing smaller models.

Existing MLOps stacks may not be ready for LLM model parallelism.

Root cause: Models cannot fit in one GPU and data parallelism alone is insufficient.

Resolution: Add Kubernetes/PyTorchJob model-parallel training with DeepSpeed ZeRO and shared recipes.

Serving large LLMs directly can be too slow or expensive for e-commerce.

Root cause: Pre-LLM serving was CPU-only, and multi-billion-parameter models require GPU serving and careful latency/throughput design.

Resolution: Use Triton for real-time GPU serving, distill smaller models, export embeddings, and use batch/nearline inference where possible.

Hybrid multi-region infrastructure adds consistency overhead.

Root cause: Cloud and on-prem clusters need consistent storage, networking, and developer experience.

Resolution: Accept the complexity where GPU availability and training-cost benefits outweigh the operational burden.

Evaluation

Coupang evaluates models and infrastructure through application performance comparisons during training, price-to-performance GPU benchmarking, model-size ROI experiments, quality checks for weak labels, throughput experiments with vLLM, and production quality indicators across search, discovery, catalog, operations, ads, relevance, and categorization.

Metrics: model performance on the application · compute budget · dataset size · model-size ROI · price-to-performance ratio by GPU and model line · weak-label quality checks · relevance model quality · categorization precision gains · vLLM throughput improvement · resource efficiency of GPU training and inference stacks

Outcomes

• Coupang deployed LLM patterns across image/language understanding, weak-label generation, product categorization, attribute extraction, search, discovery, catalog, operations, and ads quality.
• LLM-generated weak labels improved relevance model quality and helped with label scarcity in Korean and Mandarin.
• A unified LLM-powered categorizer replaced multiple category-specific models and produced precision gains across most categories.
• The platform supports real-time Triton serving, Ray/vLLM batch inference, nearline inference, distillation, embeddings, fine-tuning, and continued pretraining.
• vLLM experimentation produced nearly 20x throughput improvement in multiple workloads.
• Coupang continues to invest in larger-model training and GPU training/inference resource efficiency.

The case covers Coupang's shopping/eats/play surfaces, over 100 fulfillment centers, millions of products, Korean/Mandarin/English data, favored 3B-20B models, explicit 70B+ versus sub-10B ROI checks, A100-80GB training for >1B mixed-precision models, A10G-24GB testing, APAC/US cloud clusters, on-prem A100/H100 clusters, Triton on AWS EKS, and nearly 20x vLLM throughput improvements.