LLM-Generated Entity Profiles for Personalized Food Delivery Platform

doordash · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Food delivery personalization

DoorDash needed interpretable representations of consumers, merchants, and items for personalization, recommendations, support, and downstream ML, where dense embeddings were compact but opaque and hard to debug.

Constraints:

• Traditional embeddings are compute-friendly but difficult for humans to interpret or edit.
• Profiles must cover consumers, merchants, and items with different input data and facets.
• LLM-generated profiles need batch infrastructure, cost control, serving APIs, versioning, evaluation, and iteration loops to be reliable in production.
• Factual fields should not be inferred by LLMs when code or SQL can compute them directly.

Architecture

DoorDash built batch LLM pipelines that generate human-readable JSON profiles for consumers, merchants, and items. Data preparation gathers order history, menus, reviews, ratings, images, descriptions, and customization options; PySpark prepares prompts; Fabricator submits bulk LLM requests through Portkey files; batch outputs are downloaded, post-processed, stored in a low-latency feature store or database, and exposed through versioned APIs. Evaluation uses offline LLM-as-judge scoring and online product feedback loops.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Schema_definition["Schema definition"] --> Data_preparation_job["Data preparation job"]
  Data_preparation_job --> Amazon_S3["Amazon S3"]
  Amazon_S3 --> Prompt_generation_job["Prompt generation job"]
  Prompt_generation_job --> Batch_job_submission_job["Batch job submission job"]
  Batch_job_submission_job --> LLM_profile_generator["LLM profile generator"]
  LLM_profile_generator --> Batch_result_download_job["Batch result download job"]
  Batch_result_download_job --> Post_processing_jobs["Post-processing jobs"]
  Post_processing_jobs --> Feature_store["Feature store"]
  Feature_store --> Versioned_profile_API["Versioned profile API"]
  Versioned_profile_API --> Profile_powered_applications["Profile-powered applications"]
  Evaluation_loop["Evaluation loop"] --> LLM_profile_generator
  Versioned_profile_API --> Evaluation_loop

Components

Data flow

Structured source data is gathered into Amazon S3, prompt-generation jobs create per-entity prompts in PySpark, batch submission uploads Portkey files and tracks batch IDs in S3, batch download retrieves outputs about a day later, post-processing appends non-inferential fields, and downstream features retrieve versioned profiles from low-latency storage.

Why this architecture

DoorDash separates factual extraction from narrative synthesis: code and SQL compute objective facts, while LLMs synthesize human-readable preference, ambiance, and flavor narratives. Batch processing cuts cost and makes millions-scale profile generation practical.

Models used: LLMs · reasoning models with medium reasoning effort · LLM-as-a-judge · DNN embeddings · semantic IDs · PySpark · Fabricator · Portkey

Scale

• DoorDash can generate profiles for millions of consumers, merchants, and items.
  “We can generate profiles for millions of consumers, merchants, and items in a fraction of the time it would take to do so manually.”article, stated
• Batch processing cuts per-profile spend by roughly 50%.
  “Running in batch cuts per-profile spend by roughly 50%”article, stated
• The pipeline regularly pushes millions of records within an hour on a modest cluster.
  “We regularly push millions of records within an hour on a modest cluster size.”article, stated
• Batch output is typically promised for completion within 24 hours.
  “Usually, a batch job is promised for completion within 24 hours.”article, stated

Alternatives considered (roads not taken)

Use dense embeddings alone for entity representation

Rejected because: Embeddings are compact and efficient but opaque, hard to debug, and difficult to explain to consumers and merchandisers.

Use LLMs for all fields including objective facts

Rejected because: DoorDash's guiding principle is to use code for facts and reserve LLMs for narrative synthesis.

Issue one LLM request per entity

Rejected because: Bulk batch submission amortizes latency, network overhead, and container start-up across thousands of rows.

Patterns

semantic-caching (serving)

DoorDash generates profiles in bulk batch jobs, stores them in low-latency systems, and exposes a versioned retrieval API rather than regenerating profiles on demand.

llm-as-judge (evaluation)

DoorDash uses LLM-as-a-judge offline to score profile quality against source data, generated profile text, and a rubric.

eval-driven-development (evaluation)

Profile quality is managed through offline comparisons, task-specific evaluation, online feedback, A/B experiments, and prompt updates.

Tradeoffs

Use narrative profiles alongside embeddings.

Chose LLM-generated human-readable profiles plus traditional embeddings over Dense embeddings alone.

Rationale: Profiles add explainability, editability, prototyping, and human-readable insight while still feeding next-generation ML features.

Run profile generation as batch jobs.

Chose Bulk batch generation through Fabricator and Portkey files over Per-entity online LLM requests.

Rationale: Batch processing reduces per-profile cost and supports millions of records quickly.

Gotchas (what bit them)

Embeddings are hard to explain or debug.

Root cause: Dense vectors are compact mathematical representations but opaque to humans.

Resolution: Generate natural-language profiles that preserve semantic nuance and can be read, edited, or used for explainable recommendations.

LLM inference should not be used for objective fields.

Root cause: Facts such as average spend, order frequency, and explicit tags are better computed directly.

Resolution: Use code or SQL for factual extraction and LLMs only for narrative synthesis.

Static generated profiles can become stale.

Root cause: Consumer preferences and profile quality change over time and with feedback.

Resolution: Build evaluation and feedback loops, treat prompts as code, and eventually fine-tune on validated examples.

Evaluation

DoorDash plans offline profile-version comparison, model-based LLM-as-judge scoring with rubrics, task-specific profile evaluation, online A/B experiments, explicit profile feedback, and closed-loop prompt iteration.

Metrics: offline profile quality score · task-specific evaluation · online A/B experiment results · explicit feedback · profile version comparisons

Outcomes

• DoorDash profiles now cover consumers, merchants, and items.
• Consumer and merchant profile pipelines run in Fabricator batch jobs.
• The item profile generation job is planned to be productionized by the end of 2025.
• Profiles enable explainable recommendations, editable preferences, conversational search, and richer ML inputs.

DoorDash states that the pipeline regularly pushes millions of records within an hour and cuts per-profile spend by roughly 50% in batch.