Enhancing Ads Engagement with Multi-gate Mixture-of-Experts and Knowledge Distillation

pinterest · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Ads engagement ranking

Pinterest needed to scale ads engagement ranking quality when simply adding layers and parameters to DCNv2-style models no longer produced proportional metric gains, while short data-retention windows made new model comparisons unfair.

Constraints:

Ads engagement data volume is huge but retained only for a relatively short period.
Serving multiple experts can increase infrastructure cost.
Online comparisons are unfair when old production training data is no longer available.
Incremental training can overfit if distillation is kept after promotion.
Even small offline accuracy gains matter in the ads engagement model.

Architecture

Pinterest combined a Multi-gate Mixture-of-Experts architecture with knowledge distillation from the production model. MMoE routes inputs through multiple expert subnetworks and task-specific gates, while distillation adds a prediction-difference loss during batch training so the experimental model can learn from behavior represented by the old production model even after old training data has been deleted. Serving cost is managed with mixed precision inference and lightweight gate layers.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  ads-engagement-data["Ads engagement data"] --> batch-training-pipeline["Batch training pipeline"]
  production-dcnv2-model["Production DCNv2 model"] --> knowledge-distillation-loss["Knowledge distillation loss"]
  batch-training-pipeline --> knowledge-distillation-loss
  batch-training-pipeline --> mmoe-experts-and-gates["MMoE experts and gates"]
  knowledge-distillation-loss --> mmoe-experts-and-gates
  mmoe-experts-and-gates --> mixed-precision-serving["Mixed precision serving"]
  mixed-precision-serving --> ads-ranking-service["Ads ranking service"]
  ads-ranking-service --> ads-engagement-data

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Ads engagement data`	Large-volume training data for ads-user matching and engagement actions.	`Batch training pipeline`
`Production DCNv2 model`	Existing production model used as baseline and distillation teacher.	`Knowledge distillation loss`
`Batch training pipeline`	Trains the experimental model with labels and production-model prediction targets.	`MMoE experts and gates`, `Knowledge distillation loss`
`Knowledge distillation loss`	Transfers behavior from the production model to mitigate deleted-data gaps.	`MMoE experts and gates`
`MMoE experts and gates`	Specialized expert networks and gate layers for multi-task ads engagement modeling.	`Mixed precision serving`
`Mixed precision serving`	Reduces inference latency and infrastructure cost for serving the larger model.	`Ads ranking service`
`Ads ranking service`	Serves production ad-user matching and engagement ranking.	`Ads engagement data`

Data flow

Historical ads engagement examples train a larger MMoE experimental model. During batch training, labels contribute cross-entropy loss and the existing production model contributes distillation targets. After offline and online evaluation, the model is promoted without retaining distillation in incremental training, and mixed precision plus simple gates reduce serving cost.

“we distill knowledge from existing production models to experimental models during the batch training stage to make the experimental models converge to a better state.”article, stated

“on top of the standard cross entropy loss calculated using binary labels, we add a new loss to calculate the prediction differences between the experimental model and the production model.”article, stated

Why this architecture

MMoE gave Pinterest a way to scale model capacity through specialized experts instead of simply adding parameters to DCNv2. Knowledge distillation addressed the short-retention training-data gap, and mixed precision plus lightweight gates kept serving costs from overwhelming the model gains.

“simply adding more layers and more parameters cannot bring proportional metric gain.”article, stated

“By applying this mixed precision inference, we observe 40% inference latency reduction in benchmarking and this latency reduction translates to a significant amount of infrastructure cost.”article, stated

Models used: MMoE · DCNv2 · MaskNet · FinalMLP · knowledge distillation

Scale

Ads engagement data volume is huge.
“In the ads engagement modeling world, the volume of data is huge.”article, stated
Training data is usually retained for a few months to a year.
“retain the data for a relatively short period of time, usually a few months to a year.”article, stated
Mixed precision reduced benchmark inference latency by 40%.
“we observe 40% inference latency reduction in benchmarking”article, stated
A 0.1% offline accuracy improvement is significant for this ranking model.
“0.1% offline accuracy improvement in the Engagement ranking model is considered to be significant.”article, stated

Alternatives considered (roads not taken)

Make the DCNv2-style model deeper or wider.

Rejected because: Simply adding layers and parameters did not bring proportional metric gains.

“simply adding more layers and more parameters cannot bring proportional metric gain.”article, stated

Use more or more sophisticated experts without cost constraints.

Rejected because: ROI decreased as more experts were added, and Pinterest chose the expert mix based on metric-infrastructure tradeoffs.

“the return of investment (ROI) is decreasing with more experts added and for our use case DCNv2 has the highest ROI.”article, stated

Keep distillation loss during incremental training.

Rejected because: Experiments showed significant overfitting, so distillation was removed after batch training.

“we observe significant overfitting if we keep the distillation loss in incremental training and thus we decide to remove distillation loss in the incremental training stage.”article, stated

Patterns

`model-distillation` (modeling)

The experimental ads model learns from production-model predictions to compensate for deleted historical data.

“knowledge distillation can kick in here to make sure the retrained production model can have on-par offline and online performance.”article, stated

`cost-aware-routing` (serving)

MMoE selectively activates relevant experts and uses simple gates to limit compute overhead.

“dynamically allocating computational resources to different sub-networks (experts) based on the input data, ensuring that only the most relevant experts are activated”article, stated

`latency-optimization` (serving)

Mixed precision inference reduced latency and infrastructure cost for serving the larger model.

“From experiments, we verified that mixed precision inference has nearly zero impact on model offline performance.”article, stated

`batch-offline-processing` (training)

Knowledge distillation is applied during batch training, not incremental production training.

“Applying the knowledge distillation loss in the batch training stage seems undoubtedly correct, but the story is different for the incremental training stage.”article, stated

Tradeoffs

Expert architecture

Chose A carefully selected MMoE expert mix with DCNv2 as highest-ROI expert. over Adding more experts or more advanced expert architectures indiscriminately..

Rationale: Pinterest optimized for metric gain against infrastructure cost.

“we chose the most suitable combination of experts via careful consideration of metrics-infrastructure cost tradeoff.”article, stated

Distillation stage

Chose Apply distillation in batch training only. over Continuing distillation through incremental training or self-distillation after promotion..

Rationale: Keeping distillation in incremental training caused significant overfitting.

“when the experimental model is promoted to production, the question becomes harder to answer: should a model distill from itself?”article, stated

Gotchas (what bit them)

More parameters did not buy proportional ads-ranking gains.

Root cause: DCNv2-style scaling hit diminishing returns for the engagement model.

Resolution: Use MMoE experts and gates to learn specialized patterns for different tasks.

“The MMoE model architecture is a potential solution to help our engagement model learn more complicated patterns and relationships between users and their ads engagement actions”article, stated

Short data retention made new model comparisons unfair.

Root cause: The current production model's training data may no longer be available when a new model is ready for online testing.

Resolution: Distill from the production model so the experimental model learns from the old model's behavior.

“the training data of the current production model is no longer available, making the comparison between the new experimental model and the production model unfair.”article, stated

Serving MMoE can raise infrastructure cost.

Root cause: Multiple experts are introduced into the model.

Resolution: Use mixed precision inference and lightweight gates.

“Serving an MMoE style model would bring noticeable infrastructure cost increase since multiple experts are introduced”article, stated

Evaluation

Pinterest evaluated offline and online metrics for RelatedPins and Search against a production DCNv2 baseline, treating 0.1% offline accuracy improvement as significant.

Metrics: Offline accuracy improvement · Online metrics movement · Inference latency reduction · Infrastructure cost impact · RelatedPins and Search view types

“In this section, we show some offline and online results [8] for the MMoE model on different view types (RelatedPins and Search).”article, stated

“0.1% offline accuracy improvement in the Engagement ranking model is considered to be significant.”article, stated

Outcomes

MMoE with knowledge distillation significantly improved online and offline metrics.
Knowledge distillation mitigated performance gaps from deleted training data.
Mixed precision achieved 40% benchmark inference-latency reduction with near-zero offline impact.
Pinterest kept distillation out of incremental training to avoid overfitting.

The system operates on huge ads engagement data with retention of a few months to a year; 0.1% offline accuracy is significant and mixed precision reduced latency by 40%.

“the MMoE model architecture with knowledge distillation increases both online and offline metrics very significantly.”article, stated