Large Foundation Model for Unified Recommendation and Ranking at Scale

linkedin · Tech · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Unified recommendation and ranking at platform scale

LinkedIn wanted one foundation model to solve many personalization and ranking tasks instead of maintaining task-specific recommender models that were disjointly optimized, slow to roll out, and weak at zero-shot or cold-start generalization. The production constraint was severe: recommendation systems require very high QPS and sub-second latency.

Constraints:

• existing recommender models were trained for specific tasks and rolled out one by one
• new surfaces should be handled zero-shot without collecting full task-specific data and training a new model
• cold-start users may have fewer than five or fewer than 100 interactions
• personalization prompts must encode user profile, user history, and candidate items
• the quality-maximized model has 150 billion parameters and cannot serve online directly
• production recommendation systems can require tens of thousands of QPS and roughly 400-500 ms latency
• compression must preserve ranking calibration and generalization

Architecture

LinkedIn converts user profiles, histories, interactions, instructions, and candidate items into prompts, trains a large personalization foundation model through upcycling, continuous pretraining, fine-tuning, instruction tuning, and alignment, then compresses it for serving. The quality-maximized 150B model acts as the teacher. Production uses gradual distillation through intermediate sizes such as 8B, 3B, and 1B, gradual pruning, mixed-precision quantization with FP32 kept for the LM head, and attention sparsification/multi-item scoring so candidates attend to user history but not each other. The team built automation so experiments push results to spreadsheets and developers can vary quantization or model parameters end-to-end.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  User_profile_and_history["User profile and history"] --> Promptification_layer["Promptification layer"]
  Candidate_items["Candidate items"] --> Promptification_layer
  Promptification_layer --> Foundation_model_training_pipeline["Foundation model training pipeline"]
  Foundation_model_training_pipeline --> 150B_teacher_model["150B teacher model"]
  150B_teacher_model --> Stepwise_distillation_pipeline["Stepwise distillation pipeline"]
  Stepwise_distillation_pipeline --> Gradual_pruning_pipeline["Gradual pruning pipeline"]
  Gradual_pruning_pipeline --> Production_sized_student_model["Production-sized student model"]
  Mixed_precision_quantization["Mixed precision quantization"] --> Production_sized_student_model
  User_profile_and_history --> Sparse_attention_scorer["Sparse attention scorer"]
  Candidate_items --> Sparse_attention_scorer
  Sparse_attention_scorer --> Production_sized_student_model
  Production_sized_student_model --> Recommendation_serving_system["Recommendation serving system"]
  150B_teacher_model --> Benchmark_suite["Benchmark suite"]
  Production_sized_student_model --> Benchmark_suite
  Automation_system["Automation system"] --> Foundation_model_training_pipeline
  Automation_system --> Benchmark_suite

Components

Data flow

LinkedIn gathers user profile information, user interaction history, and candidate items, then promptifies them into an instruction-style LLM input. Training starts from an open-source model, uses upcycling to tune size/quality/throughput, then applies continuous pretraining, fine-tuning, instruction fine-tuning, and alignment. The resulting 150B-quality teacher is evaluated on cold-start and out-of-domain recommendation tasks. Serving compresses the teacher step by step into smaller models, applies pruning, mixed precision quantization, sparse attention, and multi-item scoring, then benchmarks latency and throughput improvements for production recommendation workloads.

Why this architecture

The architecture separates quality maximization from production serving. A large teacher captures holistic user journey, zero-shot behavior, cold-start in-context learning, and new-domain generalization. Because that model cannot meet high-QPS low-latency serving constraints, LinkedIn gradually distills, prunes, quantizes, sparsifies attention, and automates experiments until the student model preserves enough quality while meeting production cost and latency needs.

Models used: 150B personalization foundation model · 8B student model · 3B student model · 1B student model · Mixtral 7B · Mixtral 8x22B · SG lang · vLLM · Lightning

Scale

• The teacher model has 150 billion parameters.
  “a large model with 150 billion parameters that does really really well”transcript, stated
• The productionized student target includes a 3B model.
  “we go all the way down to let's say 3B model which is actually something that can be productionized”transcript, stated
• Recommendation serving can require tens of thousands of QPS.
  “Many recommendation systems have tens of thousands of the of the QPS”transcript, stated
• Recommendation serving targets around 400-500 ms latency.
  “they also require more less than a second like a 500 400 millisecond latency”transcript, stated
• Data-scaling experiments used behavior history going back six months or one year.
  “you might have a lot of data that goes back to 6 months, one year or whatever.”transcript, stated
• The team reports a 7x latency reduction.
  “we were able to reduce the latency by 7x”transcript, stated
• The team reports a 30x throughput increase per GPU.
  “increasing the throughput which is basically the number of queries that we can handle by one GPU by 30x.”transcript, stated

Alternatives considered (roads not taken)

Maintain separate task-specific recommender models

Rejected because: Task-specific models are disjointly optimized, slow to roll out, and cannot solve new tasks zero-shot.

Train only a small model from scratch

Rejected because: The speakers say small models lack enough capacity and that distillation from a larger model is necessary.

Distill directly from 150B to 1B

Rejected because: Stepwise distillation through intermediate sizes is described as more effective.

Aggressive pruning at the beginning

Rejected because: Aggressive pruning can reduce model quality; gradual pruning can preserve information.

Apply FP8 everywhere

Rejected because: The LM head needs FP32 precision so ranking probabilities stay calibrated.

Patterns

fine-tuning-adaptation (model-training)

LinkedIn adapts an open-source LLM into a personalization model with continuous pretraining, fine-tuning, instruction tuning, and alignment on promptified recommendation data.

retrieval-ranking (ranking)

The LLM foundation model scores user-item relevance for recommendation and ranking surfaces.

cost-aware-routing (serving)

The serving recipe compresses the teacher model through distillation, pruning, quantization, and sparse attention to lower latency and raise throughput.

batch-offline-processing (training)

Large model training, distillation, pruning, quantization experiments, and benchmark runs are automated offline before production serving.

eval-driven-development (evaluation)

LinkedIn uses broad benchmarking, including internal and external tasks, to check cold-start behavior, out-of-domain generalization, and instruction-following after compression.

Tradeoffs

Train one foundation model for many recommendation tasks.

Chose unified personalization foundation model over many task-specific recommendation models.

Rationale: A single model can generalize to new surfaces, support cold-start users, and reduce one-by-one model rollout work.

Use promptification for recommender data.

Chose LLM prompts containing profile, history, instruction, and candidate information over traditional task-specific feature input only.

Rationale: Promptification lets the model follow instructions, use in-context examples, and generalize across recommendation tasks.

Go big then go small.

Chose 150B teacher followed by gradual student distillation over training the small production model directly.

Rationale: The large model maximizes quality, and staged distillation transfers that behavior into a deployable model.

Score many candidate items together with sparse attention.

Chose multi-item scoring with attention masks over decoding or fully connected candidate attention.

Rationale: Candidates can share user-history context without attending to each other, improving serving efficiency.

Gotchas (what bit them)

A quality-maximized 150B recommender foundation model cannot directly serve high-QPS recommendations.

Root cause: Recommendation systems need tens of thousands of QPS and hundreds-of-milliseconds latency.

Resolution: Distill, prune, quantize, and sparsify attention into a production-sized student.

Small recommender LLMs trained directly lack quality.

Root cause: They do not have enough capacity or reasoning power for complex personalization tasks.

Resolution: Train a large teacher first, then distill gradually into smaller models.

Longer user context helps until the model fails to generalize to long context.

Root cause: Recommendation context length controls how much history the model sees, but some models degrade beyond their long-context generalization range.

Resolution: Experiment with context length and user-history ordering instead of assuming more history always helps.

Ranking probabilities can collapse under overly aggressive quantization.

Root cause: Lower-precision LM heads lose score calibration needed to distinguish recommended items.

Resolution: Use mixed precision and keep the LM head in FP32.

Smart history retrieval can conflict with serving optimizations such as KV caching.

Root cause: More complex context selection can break or complicate serving cache assumptions.

Resolution: Use chronological order when good enough and design smarter retrieval with serving constraints in mind.

Evaluation

LinkedIn evaluates the foundation model across cold-start user groups, out-of-domain tasks never seen in training, internal and external benchmark sets, instruction-following tests, compression quality, latency, and throughput. Experiments also vary data scale, model size, context length, distillation, pruning, quantization, and serving kernels.

Metrics: cold-start performance by number of interactions · out-of-domain task performance · 50 to 60 benchmarks · instruction-following benchmark quality · latency reduction · queries handled per GPU · model quality loss under pruning · ranking calibration under quantization

Outcomes

• The model showed better relative performance for users with fewer interactions compared with production models.
• The model could be on par with or beat production task-specific models on four out-of-domain tasks.
• Stepwise compression produced a production-sized model path around 3B parameters.
• Serving optimizations reduced latency by 7x.
• Serving optimizations increased throughput per GPU by 30x.
• Automation allowed developers to run end-to-end experiments by changing parameters such as quantization settings.

The transcript reports a 150B teacher, production-sized 3B models, candidate groups up to 500 items, high-QPS recommendation serving, roughly 400-500 ms latency targets, 7x latency reduction, 30x throughput improvement per GPU, and around 50-60 benchmark sets.