Foundation Model for Ads Recommendation at Scale

meta · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: large-scale ads recommendation and personalization

Meta needed an LLM-scale foundation model for ads recommendation that could learn from sparse, multimodal, cross-surface user-ad interactions, improve hundreds of latency-sensitive vertical models, and train efficiently across thousands of GPUs.

Constraints:

• billions of daily user-ad interactions contain sparse meaningful signals such as clicks and conversions
• ads data spans advertiser goals, creative formats, measurement signals, user behaviors, and multiple delivery channels
• training an LLM-scale recommender requires thousands of GPUs and efficient distributed systems
• downstream user-facing vertical models need GEM's knowledge without incurring full foundation-model inference cost
• surface-specific predictions must remain tailored to Facebook, Instagram, Business Messaging, and other contexts

Architecture

GEM is Meta's LLM-scale ads recommendation foundation model. It learns from ad content plus organic and ads engagement, separates sequence and non-sequence features, applies customized attention and cross-feature learning, uses Wukong-style stackable factorization machines, pyramid-parallel sequence processing, InterFormer cross-feature learning, and multi-domain optimization. Post-training, GEM transfers knowledge into hundreds of vertical models through direct and hierarchical distillation, student adapters, representation learning, and parameter sharing, all backed by a re-engineered distributed training stack.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ad-and-engagement-data[Ad and engagement data]
  feature-derivation-pipeline[Feature derivation pipeline]
  sequence-modeling-stack[Sequence modeling stack]
  non-sequence-interaction-stack[Non-sequence interaction stack]
  cross-feature-learning-layer[Cross-feature learning layer]
  gem-foundation-model[GEM foundation model]
  knowledge-transfer-layer[Knowledge transfer layer]
  vertical-ads-models[Vertical ads models]
  training-infrastructure[Training infrastructure]
  ad-and-engagement-data --> feature-derivation-pipeline
  feature-derivation-pipeline --> sequence-modeling-stack
  feature-derivation-pipeline --> non-sequence-interaction-stack
  sequence-modeling-stack --> cross-feature-learning-layer
  non-sequence-interaction-stack --> cross-feature-learning-layer
  cross-feature-learning-layer --> gem-foundation-model
  gem-foundation-model --> knowledge-transfer-layer
  gem-foundation-model --> training-infrastructure
  knowledge-transfer-layer --> vertical-ads-models
  knowledge-transfer-layer --> gem-foundation-model
  training-infrastructure --> gem-foundation-model

Components

Data flow

Ad and organic engagement data are converted into sequence features such as user activity history and non-sequence features such as user attributes, ad format, and creative representation. GEM models these paths with customized attention, Wukong blocks, pyramid-parallel sequence modules, and InterFormer cross-feature learning. Its foundation-model outputs are then transferred into vertical production models using distillation, representation learning, parameter sharing, and student adapters that refresh teacher outputs with recent ground truth.

Why this architecture

Meta built GEM as a foundation model because ads recommendation had become too sparse, multimodal, and cross-surface for isolated ranking models to capture efficiently. The architecture preserves long user histories, enables cross-feature and cross-domain learning, and then transfers foundation-model knowledge into smaller vertical models so production serving can benefit without paying full foundation-model inference cost.

Models used: GEM · Wukong architecture · InterFormer · domain-specific foundation models · vertical models · Student Adapter

Scale

• GEM is trained across thousands of GPUs at LLM scale.
  “built on an LLM-inspired paradigm and trained across thousands of GPUs.”article, stated
• Meta says GEM is the largest foundation model for recommendation systems in the industry.
  “It is the largest foundation model for recommendation systems (RecSys) in the industry, trained at the scale of large language models.”article, stated
• GEM delivered a 5% increase in Instagram ad conversions and a 3% increase in Facebook Feed ad conversions in Q2.
  “delivered a 5% increase in ad conversions on Instagram and a 3% increase in ad conversions on Facebook Feed in Q2.”article, stated
• Q3 architecture changes doubled the performance benefit from adding a given amount of data and compute.
  “In Q3, we made improvements to GEM’s model architecture that doubled the performance benefit we get from adding a given amount of data and compute.”article, stated
• The scalable architecture is 4x more efficient at driving ad performance gains than Meta's original ads ranking models.
  “This design improves accuracy and scales both the depth and breadth of each attention block, delivering 4× the efficiency of our previous generation of models.”article, stated
• The knowledge-transfer framework achieves 2x the effectiveness of standard distillation.
  “achieving 2x the effectiveness of standard knowledge distillation.”article, stated
• The training stack delivers 23x effective training FLOPS, 1.43x MFU, and uses 16x more GPUs.
  “The re-engineered training stack delivers a 23x increase in effective training FLOPs using 16x more GPUs while also improving efficiency.”article, stated
• GEM transfers knowledge to hundreds of user-facing vertical models.
  “GEM only delivers impact if its knowledge can be efficiently transferred to hundreds of user-facing vertical models (VMs).”article, stated
• The offline sequence infrastructure processes up to thousands of user events.
  “The new scalable offline feature infrastructure processes sequences of up to thousands of events with minimal storage cost”article, stated

Alternatives considered (roads not taken)

train recommendation surfaces in isolation

Rejected because: Isolated surface models miss valuable cross-platform signals from Meta's broader ecosystem.

treat all Meta surfaces identically

Rejected because: Facebook, Instagram, and Business Messaging have distinct objectives and behavior patterns that need domain-specific optimization.

compress user behavior sequences into compact vectors before downstream modeling

Rejected because: Compression can lose engagement signals, so GEM preserves full sequence information while still enabling cross-feature learning.

use standard knowledge distillation alone

Rejected because: Meta uses direct and hierarchical transfer plus additional techniques to achieve better transfer effectiveness across the ads model fleet.

serve full foundation-model computation in every latency-sensitive vertical model

Rejected because: Parameter sharing lets smaller vertical models reuse foundation-model representations without paying the full compute cost.

Patterns

retrieval-ranking (ranking-personalization)

GEM is a foundation model for ads ranking and recommendation, using user-ad histories and cross-surface signals to improve ad relevance and conversion predictions.

fine-tuning-adaptation (model-adaptation)

Meta uses post-training, continuous online refresh, student adapters, representation learning, and parameter sharing to adapt GEM's knowledge into downstream production models.

eval-driven-development (model-quality)

GEM's production loop uses recent ground-truth outcomes to correct stale or mismatched teacher signals before supervising vertical models.

Tradeoffs

Use a unified foundation model with domain-specific optimization.

Chose cross-surface learning with tailored predictions per surface over isolated per-surface models or one identical treatment for all surfaces.

Rationale: GEM captures shared cross-platform insights while still optimizing each surface for its own behavior and objective.

Preserve full sequence information during cross-feature learning.

Chose InterFormer parallel summarization and interleaving over compressing user behavior sequences into compact vectors.

Rationale: Preserving sequence structure avoids losing critical behavioral signals while still enabling efficient interaction learning.

Transfer foundation-model knowledge into smaller vertical models.

Chose direct and hierarchical transfer plus parameter sharing over serving the full foundation model everywhere.

Rationale: Latency-sensitive vertical models can use GEM's representations without full foundation-model compute.

Use lightweight variants during exploration.

Chose lower-cost model variants for over half of experiments over running every experiment on full-sized GEM.

Rationale: Smaller variants accelerate idea validation and reduce resource overhead before large-scale training.

Gotchas (what bit them)

Ads recommendation signals are extremely sparse despite massive interaction volume.

Root cause: Clicks and conversions are rare relative to billions of impressions, creating class imbalance and noisy learning conditions.

Resolution: Train a foundation model over large-scale cross-surface data and architecture designed to generalize across diverse users and behaviors.

Long user behavior sequences are hard for traditional recommender architectures.

Root cause: Thousands of clicks, views, and interactions create computational and storage pressure.

Resolution: Use pyramid-parallel sequence modeling and scalable offline feature infrastructure for long histories.

Compressed sequence vectors lose important engagement signals.

Root cause: Downstream compact representations can discard details from the full user journey.

Resolution: Use InterFormer to preserve full sequence information while learning cross-feature interactions.

Teacher signals can become stale or misaligned for vertical models.

Root cause: Foundation-model training and evaluation lag behind current outcomes, and surfaces have distinct objectives.

Resolution: Use a Student Adapter to refine teacher outputs with recent ground-truth data.

Scaling recommender foundation models can waste GPU capacity without system-level optimization.

Root cause: Dense and sparse components create different communication, memory, and synchronization bottlenecks at thousands-of-GPU scale.

Resolution: Use HSDP for dense parameters, two-dimensional sparse parallelism for embeddings, custom kernels, compilation, memory compression, and NCCLX collectives.

Evaluation

Meta evaluates GEM through online conversion lift, ad-performance gain per data/compute, transfer effectiveness to vertical models, training FLOPS and MFU efficiency, startup/compilation time, and scaling behavior as data, compute, and surfaces increase.

Metrics: 5% Instagram ad conversion increase and 3% Facebook Feed conversion increase in Q2 · 4x efficiency improvement over original ads ranking models · 2x effectiveness of standard knowledge distillation · 23x effective training FLOPS, 1.43x MFU, and 16x more GPUs · 5x reduced job startup time · 7x reduced PyTorch 2.0 compilation time · over half of experiments supported by lightweight model variants

Outcomes

• GEM increased ad conversions by 5% on Instagram and 3% on Facebook Feed in Q2.
• Q3 architecture improvements doubled the performance benefit from a given amount of data and compute.
• The architecture delivered 4x efficiency improvement over Meta's original ads ranking models.
• The knowledge-transfer framework reached 2x standard distillation effectiveness.
• The re-engineered training stack produced 23x effective training FLOPS while scaling to 16x more GPUs.

The system runs at LLM scale for ads recommendation, learning from billions of user-ad interactions, training across thousands of GPUs, transferring to hundreds of vertical models, and processing sequences of up to thousands of user events.