GenAI-Powered Personalized Homepage Carousels for Food Delivery

doordash · E-commerce · 2025 · source Agentic: no · promotional bias: med · summary-only (no fetched source) · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: food-delivery homepage personalization

DoorDash needed to replace a limited heuristic homepage carousel system with personalized, diverse, and relevant carousel generation that could scale to millions of users while improving merchant discovery and engagement.

Constraints:

• the previous system relied on approximately 300 manually curated carousels
• heuristic food-knowledge-graph tagging created insufficient diversity and relevance issues
• millions of personalized carousel titles made manual review infeasible
• external LLM API costs made real-time generation for every visit unattractive
• the new relevance signal had to integrate with existing production ranking models
• consumer-facing content required conservative moderation for brand safety

Architecture

DoorDash built a five-stage GenAI personalization pipeline for homepage carousels. Spark batch jobs create prompts from consumer profiles and part-of-day context, call LLMs to generate personalized carousel titles and metadata, moderate titles with a three-model LLM jury, embed carousel metadata and merchant/dish profiles into a shared semantic space, retrieve stores and dishes with GPU exact KNN inside delivery-radius masks, and block-rerank candidates by blending existing engagement ranker scores with embedding similarity.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  consumer-profiles["Consumer profiles"] --> spark-batch-generation["Spark batch generation"]
  spark-batch-generation --> llm-carousel-generator["LLM carousel generator"]
  llm-carousel-generator --> llm-moderation-jury["LLM moderation jury"]
  llm-moderation-jury --> carousel-embedding["Carousel embedding"]
  llm-carousel-generator --> carousel-embedding
  merchant-and-dish-embeddings["Merchant and dish embeddings"] --> gpu-exact-knn-retrieval["GPU exact KNN retrieval"]
  delivery-radius-masks["Delivery-radius masks"] --> gpu-exact-knn-retrieval
  carousel-embedding --> gpu-exact-knn-retrieval
  gpu-exact-knn-retrieval --> block-reranker["Block reranker"]
  existing-engagement-ranker["Existing engagement ranker"] --> block-reranker
  block-reranker --> homepage-carousel-surface["Homepage carousel surface"]
  homepage-carousel-surface --> a-b-testing["A/B testing"]

Components

Data flow

Consumer profiles and time-of-day context feed Spark batch jobs that call LLMs to generate carousel titles and metadata. The titles are moderated by three LLM jurors, then concatenated with metadata and embedded. Merchant and dish profiles are also embedded. At serving time, KNN retrieval finds relevant stores inside a user's delivery radius and relevant dishes within stores, then the existing ranking framework partitions candidates into blocks and reranks each block using both engagement score and embedding similarity before showing homepage carousels.

Why this architecture

The design balances personalization quality, cost, safety, and production integration. Batch LLM generation controls API cost; metadata expansion makes short titles retrievable; exact GPU KNN preserves retrieval quality and latency; a moderation jury blocks unsafe titles; and block reranking lets DoorDash add embedding relevance without replacing its existing engagement ranker.

Models used: LLMs · LLM text embedding models · Spark · GPU exact KNN · existing carousel serving framework · store ranking models · food knowledge graph

Scale

• The previous system had approximately 300 manually curated carousels.
  “The system replaced a heuristic-based approach that relied on approximately 300 manually curated carousels”summary, stated
• The new pipeline was designed for millions of users globally.
  “scalability to handle millions of users globally”summary, stated
• Moderation uses three different LLMs and blocks a title if any one flags it.
  “The moderation system prompts three different LLMs with review criteria”summary, stated

  “if any single juror LLM flags a title as problematic, it is automatically blocked.”summary, stated
• The moderation approach achieved 95% recall on problematic titles.
  “This conservative approach achieves 95% recall on detecting problematic titles.”summary, stated
• Offline store-relevance precision@10 improved from 68% to 85%.
  “achieving improvement from 68% to 85% precision@10 through refinements to prompts and retrieval strategies.”summary, stated
• Online A/B tests launched in San Francisco and Manhattan.
  “DoorDash launched the new content system in two major submarkets: San Francisco and Manhattan.”summary, stated
• A/B tests showed double-digit click-rate improvements.
  “The A/B tests showed double-digit improvements in click rates”summary, stated

Alternatives considered (roads not taken)

Heuristic food knowledge graph with curated carousels

Rejected because: The earlier system had insufficient diversity, broad and impersonal concepts, and relevance issues from suboptimal tagging.

Real-time carousel generation per homepage visit

Rejected because: Batch generation was chosen as a cost-effective strategy because external LLM API calls are expensive at global-user scale.

Short title embeddings alone

Rejected because: Brief carousel titles such as 'Hearty wraps' lacked enough retrieval context, so the LLM also generated personalized metadata aligned to merchant profiles.

Approximate nearest neighbor retrieval

Rejected because: DoorDash used exact GPU KNN, accepting higher infrastructure cost for retrieval quality and low latency.

Replace the existing engagement ranker

Rejected because: DoorDash used block reranking to incorporate embedding similarity incrementally without rebuilding the production ranking system.

Patterns

batch-offline-processing (LLMOps cost control)

DoorDash pre-generates personalized carousels in Spark batch jobs instead of calling LLMs in real time for every homepage visit.

llm-as-judge (content safety)

The moderation layer prompts three LLMs as independent jurors and applies a veto rule to block any title flagged as problematic.

rag-retrieval (semantic retrieval)

Carousel titles plus metadata and merchant/dish profiles are embedded into the same semantic space so KNN retrieval can populate carousels with relevant stores and dishes.

retrieval-ranking (recommendation ranking)

DoorDash retrieves stores and dishes by embedding similarity, then reranks within blocks using both existing engagement ranking and embedding relevance.

eval-driven-development (quality evaluation)

DoorDash uses internal panels for subjective carousel quality, third-party labelers for store relevance, precision@K monitoring, and A/B tests for online business impact.

cost-aware-routing (architecture tradeoff)

The system uses cheaper batch LLM generation where freshness can lag, but spends on GPU exact KNN where retrieval quality and low latency are business-critical.

human-in-the-loop-gate (prompt quality)

Internal user panels score generated carousels and feed prompt refinement for repetition, specificity, diversity, and relevance.

Tradeoffs

LLM generation timing

Chose offline Spark batch generation over real-time LLM calls per homepage visit.

Rationale: Batch generation controls external LLM API cost at millions-of-users scale, accepting less freshness.

Retrieval method

Chose exact KNN on GPU over approximate nearest neighbor search.

Rationale: DoorDash accepted GPU cost to preserve retrieval quality and low latency for business-critical personalization.

Content moderation policy

Chose multi-LLM veto moderation over manual review or majority-vote leniency.

Rationale: A conservative veto prioritizes brand safety and high recall over moderation cost and false positives.

Ranking integration

Chose block reranking with engagement and similarity scores over fully replacing the baseline ranker or ignoring embedding relevance.

Rationale: Block reranking reduces implementation risk while making the new relevance signal usable before retraining the ranker.

Gotchas (what bit them)

Short generated titles are poor retrieval queries by themselves.

Root cause: Titles such as 'Hearty wraps' are too brief to create useful embeddings without extra context.

Resolution: Have the LLM generate auxiliary metadata aligned to merchant profile fields, personalized by user preferences and order history.

Generated consumer-facing titles need scalable moderation.

Root cause: Manual review cannot handle millions of unique carousel titles.

Resolution: Use three LLM jurors with a conservative veto rule to block problematic titles.

Existing rankers may not know how to use new embedding-similarity features.

Root cause: The existing engagement ranker had not been trained on the new embedding similarity scores.

Resolution: Use block reranking to blend ranker score and embedding similarity within local blocks while retaining the production ranker.

Batch generation introduces freshness lag.

Root cause: Pre-generating carousels reduces LLM cost but cannot respond instantly to very recent user behavior.

Resolution: Accept the staleness tradeoff for cost-effective scale while focusing online computation on retrieval and ranking.

The company write-up omits some operational failure modes.

Root cause: The available summary notes missing disclosure about output degradation, embedding drift, retrieval monitoring, costs, regeneration cadence, and weak segments.

Resolution: Treat operational monitoring and generalization as known limitations for reviewer sampling rather than asserting full observability.

Evaluation

DoorDash combined internal human scoring for subjective carousel quality, third-party labeling for carousel-store relevance, precision@K tracking for retrieval, and online A/B tests in San Francisco and Manhattan for click, conversion, relevance, bounce, exploration, and merchant-discovery impact.

Metrics: repetition frequency · specificity · diversity · relevance · precision@10 · click rate · conversion rate · homepage relevance metrics · bounce rate · merchant discovery · merchant trials

Outcomes

• Precision@10 improved from 68% to 85% through prompt and retrieval refinements.
• A/B tests in San Francisco and Manhattan showed double-digit improvements in click rates.
• The summary reports improvements in conversion rates and homepage relevance metrics.
• The system increased merchant discovery, especially for small and mid-sized businesses.
• The architecture provides a path toward broader personalization dimensions such as affordability, speed, groceries, and fine-tuned DoorDash-specific LLMs.

The system is a production-scale marketplace personalization pipeline designed for millions of users globally. The available summary reports two-market A/B tests, double-digit click-rate improvements, precision@10 rising from 68% to 85%, and merchant-discovery impact, but it does not disclose exact LLM cost, regeneration frequency, or all segment-level results.