DoorDash Summer 2025 Intern Projects: LLM-Powered Feature Extraction and RAG Chatbot Infrastructure

doordash · E-commerce · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Food-delivery operations and internal chatbot infrastructure

DoorDash needed to reduce manual operational review for never-delivered orders and give teams reusable infrastructure for RAG chatbots instead of fragmented embeddings, spreadsheets, and one-off bot implementations.

Constraints:

• DoorDash processes about 2 billion annual orders, making manual ND review expensive and slow.
• Never-delivered order analysis requires extracting precise facts from customer-Dasher conversations.
• Rare positive labels created severe class imbalance for some classification questions.
• Teams lacked centralized knowledge-base management and a unified API for launching chatbots.
• Bot behavior had to be configurable for successful retrievals and fallback cases.

Architecture

DoorDash describes two LLMOps systems from intern projects. The first is an in-house DistilBERT-based sequence classifier that processes Dasher-consumer conversations and outputs features for the downstream never-delivered order model. It handles class imbalance through rare-case oversampling and uses F1 as the primary metric. The second is a centralized Knowledge Base Management Service plus Chatbot-as-a-Service: teams create isolated vector collections, upload articles for embedding and metadata storage, configure prompts/fallbacks, and call GetRagResult for retrieval-augmented answers.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  customer-dasher-conversations[Customer Dasher conversations] --> nd-feature-classifier[ND feature classifier]
  nd-feature-classifier --> downstream-nd-model[Downstream ND model]
  knowledge-base-management-service[Knowledge Base Management Service] --> vector-db-collections[Vector DB collections]
  knowledge-base-management-service --> getragresult-api[GetRagResult API]
  vector-db-collections --> getragresult-api
  prompt-configuration[Prompt configuration] --> getragresult-api
  getragresult-api --> team-chatbot-interface[Team chatbot interface]

Components

Data flow

For ND review, customer-Dasher conversations enter the language model, which walks through the resolution decision process and emits features to the downstream ND model. For chatbots, teams upload knowledge articles through KB APIs; the platform generates embeddings and metadata, GetRagResult retrieves relevant articles from an isolated collection, combines them with conversation history and a prompt, and returns a context-aware answer plus referenced articles.

Why this architecture

The ND classifier uses a lightweight task-specific model because precise binary feature extraction outperformed general-purpose reasoning models and reduced latency. The chatbot platform centralizes ingestion, embeddings, retrieval, prompts, and references because scattered spreadsheets plus vector IDs did not scale across teams.

Models used: Meta Llama 3 70B · fine-tuned Meta Llama 3 70B · Meta Llama 3 8B · fine-tuned Meta Llama 3 8B · DistilBertForSequenceClassification · embedding models

Scale

• DoorDash has about 2 billion annual orders.
  “With about 2 billion orders annually through DoorDash, every ND order can negatively impact overall revenue and user trust.”article, stated
• Some rare ND labels represented only about 1% of orders.
  “only about 1% of ND orders had a “yes” response to the question “Did the Dasher attempt to contact the consumer but did not receive a response?””article, stated
• The balanced DistilBERT model achieved F1 0.8289 and latency 0.0936 seconds.
  “Classification (with balanced data)0.8630 0.7975 0.8289 0.9870 0.0936”article, stated
• A live example had inference time of 0.159 seconds.

  "Inference time: 0.159 seconds

Prediction: [1]"

— article, stated

Alternatives considered (roads not taken)

Manual customer-support review for ND labeling.

Rejected because: Manual review was slow, expensive, and limited resolutions to a few cases each day.

Use general-purpose Llama 3 reasoning models for ND classification.

Rejected because: The classification task needed precise event detection, and DistilBERT achieved better F1 and latency.

Let each team manage chatbot embeddings and metadata separately.

Rejected because: The spreadsheet/vector-ID workflow was error-prone and unscalable.

Build one-off chatbot infrastructure per team.

Rejected because: DoorDash lacked a plug-and-play unified API, so the team built reusable Chatbot-as-a-Service.

Patterns

fine-tuning-adaptation (model-operations)

DoorDash trained a task-specific DistilBERT sequence classifier for ND feature extraction instead of relying on general-purpose Llama reasoning models.

data-quality-loop (data)

The ND model oversampled rare positive cases and used F1 to avoid majority-class collapse on skewed labels.

rag-retrieval (retrieval)

The Chatbot-as-a-Service platform retrieves relevant knowledge-base articles and generates context-aware answers through GetRagResult.

structured-output-contract (data)

The ND classifier outputs decision-process answers as downstream model features, while KB APIs standardize article ingestion and metadata storage.

Tradeoffs

ND model family selection

Chose Fine-tuned DistilBERT sequence classification. over Fine-tuned or prompted Llama 3 variants..

Rationale: DistilBERT better matched the precise binary classification task and had better F1 and latency.

Platform chatbot infrastructure

Chose Centralized KB management plus generalized Chatbot-as-a-Service. over Team-specific bot stacks and fragmented KB workflows..

Rationale: A shared platform supports reusable onboarding, isolated collections, configurable embeddings, prompts, fallbacks, and references.

Per-bot isolation

Chose Dedicated vector collections per bot. over A single shared schema for all chatbot knowledge bases..

Rationale: Isolated collections preserve schema independence and future embedding customization.

Gotchas (what bit them)

Rare ND labels were ignored without balancing.

Root cause: Some positive cases appeared in only about 1% of the dataset, so unbalanced models predicted the majority class.

Resolution: DoorDash oversampled rare yes cases and optimized with F1.

General-purpose reasoning models underperformed on the narrow classification task.

Root cause: The production task was precise event detection, not broad text generation.

Resolution: DoorDash selected a fine-tuned DistilBERT classifier with better F1 and latency.

Separate spreadsheets and vector stores made chatbot KB management fragile.

Root cause: Teams had to manually connect article metadata in spreadsheets to vector IDs.

Resolution: DoorDash built centralized KB APIs and per-bot vector collections with stored metadata.

Evaluation

DoorDash compared multiple Llama and DistilBERT variants on precision, recall, F1, accuracy, and latency; the chatbot platform is evaluated through production readiness, retrieval references, and team onboarding.

Metrics: DistilBERT with balanced data: precision 0.8630, recall 0.7975, F1 0.8289, accuracy 0.9870, latency 0.0936 seconds · Fine-tuned Llama 3 70B: F1 0.6087 and latency 0.3275 seconds · Live example inference time: 0.159 seconds · About 2 billion annual DoorDash orders

Outcomes

• The ND feature extractor achieved higher F1 and lower latency than the tested Llama variants.
• Automation lets agents focus on more complex work rather than repetitive case reviews.
• The chatbot platform supports self-serve onboarding for RAG-powered bots across teams.
• The shared RAG API returns referenced articles for transparency and guardrail/audit use cases.

The case covers 2 billion annual orders, rare 1% positive labels, sub-second model latency, and a generalized platform designed to support many more bots across DoorDash.