Small Language Models for Real-Time Ad Relevance Prediction

doordash · E-commerce · 2026 · source Agentic: no · promotional bias: low · summary-only (no fetched source) · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Sponsored search ad relevance prediction

DoorDash needed to filter irrelevant sponsored search results in real time, where keyword retrieval and personalization could promote semantically wrong ads despite overlapping tokens.

Constraints:

• Keyword-based retrieval optimized for recall could surface irrelevant ads like chips for a salt query.
• Personalization could amplify poor relevance if a user historically bought the wrong item category.
• Human labeling alone cannot cover constantly changing query-item combinations at catalog scale.
• The production model needed low-latency serving in the ad auction pipeline.

Architecture

DoorDash uses a teacher-student relevance architecture: roughly 700,000 human-labeled query-item pairs define a 0-1-2 relevance task, a fine-tuned closed-source LLM teacher generates offline labels for production traffic pairs, a DistilBERT bi-encoder student with shared weights and 64-dimensional cached embeddings learns the teacher's judgments, and online serving fetches precomputed query/item embeddings, scores them with a bilinear layer, and applies the relevance filter before the ad auction while quality scoring runs in parallel.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Human relevance labels"] --> B["LLM teacher"]
  C["Production traffic pairs"] --> B
  B --> D["Teacher-generated labels"]
  D --> E["DistilBERT bi-encoder"]
  E --> F["Offline embedding job"]
  F --> G["Embedding key-value store"]
  G --> H["Online relevance scorer"]
  I["Quality scorer"] --> J["Ad auction"]
  H --> J

Components

Data flow

Human labels train a relevance teacher. The LLM teacher labels query-item pairs from six months of production traffic offline. A DistilBERT bi-encoder student trains on those labels, embeddings are computed offline and cached in a key-value store, and online serving performs lightweight bilinear relevance scoring before auction ranking.

Why this architecture

The architecture separates expensive semantic judgment from online inference: an LLM teacher can label at offline scale, while a cached-embedding DistilBERT student gives the ad serving path low latency and keeps irrelevant ads from entering the auction.

Models used: closed-source LLM teacher · DistilBERT · fine-tuned GPT evaluation models · gradient boosted decision tree baseline

Scale

• DoorDash started with approximately 700,000 human-labeled query-item pairs.
  “human annotators labeling approximately 700,000 query-item pairs”summary, stated
• The LLM teacher labels query-item pairs from six months of production traffic.
  “query-item pairs extracted from six months of real production traffic.”summary, stated
• The final production model uses 64-dimensional embeddings.
  “64-dimensional embeddings struck the optimal balance between relevance quality, dimensionality reduction, and serving latency.”summary, stated
• The production checkpoint reached 97% validation accuracy.
  “The final production checkpoint was selected based on validation accuracy, with the best-performing model reaching 97% accuracy.”summary, stated
• Online A/B testing showed a 5.2% relative Precision@2 gain over the prior GBDT model.
  “achieved a 5.2% relative gain in Precision@2 over the incumbent gradient boosted decision tree production model during online A/B testing.”summary, stated

Alternatives considered (roads not taken)

Keyword-based ad retrieval alone

Rejected because: Keyword overlap can return semantically irrelevant sponsored items.

Use personalization to fix relevance

Rejected because: Personalization can reinforce irrelevant matches when user history favors the wrong category.

Run the LLM teacher online

Rejected because: The teacher is for offline supervision; online serving requires a compact low-latency model.

Matryoshka dimensionality reduction

Rejected because: DoorDash chose direct linear projection as simpler, efficient, and production-ready.

Patterns

model-distillation (model-adaptation)

DoorDash transfers offline LLM teacher judgments into a compact DistilBERT student for serving.

batch-offline-processing (infrastructure)

The LLM teacher labels production traffic pairs offline and embeddings are precomputed by cron before serving.

latency-optimization (performance)

DoorDash moves neural encoding offline and runs only lightweight cached-embedding scoring online.

llm-as-judge (evaluation)

DoorDash uses fine-tuned GPT models for scalable and consistent relevance-quality measurement.

Tradeoffs

Separate teacher and student roles.

Chose offline LLM teacher plus online SLM student over one model for both label quality and low-latency serving.

Rationale: This preserves high-quality supervision while meeting production latency constraints.

Use a bi-encoder with cached embeddings.

Chose offline query/item embeddings and online bilinear scoring over online neural encoding for every request.

Rationale: Cached embeddings move expensive computation out of the serving path.

Filter before auction.

Chose relevance as a pre-auction gate over letting bids compensate for irrelevance.

Rationale: Relevance is treated as non-negotiable business logic before auction pricing.

Gotchas (what bit them)

Recall-oriented keyword retrieval can produce obviously irrelevant ads.

Root cause: Overlapping tokens do not guarantee semantic match to search intent.

Resolution: Add a user-agnostic relevance model that filters label-0 items before the auction.

Teacher labels can scale errors as well as knowledge.

Root cause: The student learns from millions of teacher-generated labels, not only human ground truth.

Resolution: Validate with online A/B tests and maintain awareness that the student may reproduce teacher errors.

Closed-source teacher dependency creates control and reproducibility concerns.

Root cause: The LLM teacher relies on an external or closed model.

Resolution: Explore migration to fine-tuned open-source alternatives.

Filtering affects future behavioral data.

Root cause: Items removed by relevance filters are not shown, changing what engagement data is collected.

Resolution: The summary flags feedback loops as an open operational consideration, not a fully solved area.

Evaluation

DoorDash evaluates relevance with validation accuracy, three-class and binary teacher accuracy, architecture comparisons, Precision@2, NDCG@10, fine-tuned GPT evaluators, and online A/B testing against a GBDT baseline.

Metrics: three-class accuracy · binary accuracy · validation accuracy · Precision@2 · NDCG@10 · online A/B lift · serving latency

Outcomes

• The final SLM reached 97% validation accuracy.
• The online A/B test achieved a 5.2% relative Precision@2 gain over the prior GBDT model.
• The model runs relevance and quality scoring in parallel so the relevance layer adds near-zero latency.
• The filter prevents irrelevant ads from entering the auction solely because of bid strength.

The case includes 700,000 human-labeled examples, six months of production traffic for offline labels, millions of teacher labels, 64-dimensional embeddings, and distributed multi-GPU training.