AI-Powered Content Moderation at Scale: SafeChat Platform

doordash · E-commerce · 2025 · source Agentic: no · promotional bias: med · summary-only (no fetched source) · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Marketplace safety moderation

DoorDash needed to moderate high-volume chat, image, and voice communications between Dashers and consumers in real time while balancing safety, latency, cost, and false positives.

Constraints:

• Chat and image moderation must run inline before content reaches recipients
• Voice calls require reactive rather than fully preventive actions
• The fetched source is unavailable, so this extraction is grounded only in the ZenML summary

Architecture

SafeChat moderates DoorDash Dasher/customer text, image, and voice communications with a cost-aware cascade. The original text path used three external layers; after collecting production labels, DoorDash replaced the first two layers with an internal safeScore/unsafeScore classifier and reserved a precise external LLM for the small ambiguous/high-risk tail. Image and voice paths use separate model-selection, threshold tuning, and rollout processes.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  internal-text-classifier["internal-text-classifier"]
  precise-external-llm["precise-external-llm"]
  graduated-response-engine["graduated-response-engine"]
  image-moderation-model["image-moderation-model"]
  threshold-tuning-dataset["threshold-tuning-dataset"]
  voice-moderation-rollout["voice-moderation-rollout"]
  human-safety-agents["human-safety-agents"]
  internal-text-classifier --> precise-external-llm
  internal-text-classifier --> graduated-response-engine
  precise-external-llm --> graduated-response-engine
  graduated-response-engine --> human-safety-agents
  image-moderation-model --> threshold-tuning-dataset
  voice-moderation-rollout --> human-safety-agents

Components

Data flow

A chat message first passes through an internal model that outputs safeScore and unsafeScore. High-confidence safe messages clear immediately; suspicious messages go to the precise external LLM, which scores profanity, threats, and sexual content and can trigger graduated actions such as penalty-free cancellation. Images enter a separate vision pipeline tuned through shadow testing and threshold refinement, while voice moderation rolls out asynchronously.

Why this architecture

DoorDash used cascades because moderation must balance cost, latency, and precision at delivery-platform scale. Production labels from the first architecture became the training data for a cheaper internal classifier, while the precise external LLM remained available for the rare messages where confidence and nuance mattered most.

Models used: free moderation API · fast LLM · precise LLM · internal moderation model · granular computer vision model

Scale

• SafeChat processes millions of chat messages, hundreds of thousands of images, and voice calls daily.
  “The platform processes millions of chat messages, hundreds of thousands of images, and numerous voice calls daily to maintain safety and trust on the delivery platform.”summary, stated
• The first cascade cleared about 90% of chat messages at the free first layer.
  “By setting the threshold very low, this layer could automatically clear approximately 90% of all chat messages without incurring any costs.”summary, stated
• Only about 0.2% of messages required the most intensive precise LLM layer in the original cascade.
  “By the end of this second layer, around 99.8% of all messages were identified as safe, leaving only 0.2% requiring the most intensive analysis.”summary, stated
• The internal model was trained after accumulating about 10 million labeled data points.
  “After running the three-layer approach for several months, DoorDash accumulated approximately 10 million labeled data points from the production system.”summary, stated
• The internal model responds under 300ms for most messages, while flagged messages may take up to three seconds.
  “The internal model responds in under 300ms for most messages, with only flagged messages experiencing up to three seconds of latency.”summary, stated
• SafeChat achieved a 50% reduction in low- to medium-severity safety incidents since launch.
  “Since launch, SafeChat has achieved a 50% reduction in low to medium-severity safety incidents while maintaining low latency”summary, stated

Alternatives considered (roads not taken)

Send every message to the slowest, most precise LLM

Rejected because: The cascade reserves expensive precision for the rare ambiguous/high-risk tail while cheaper layers clear routine content.

Keep relying on external models after enough production labels existed

Rejected because: DoorDash trained an internal first-layer model from production labels to make high-volume moderation more sustainable.

Deploy the first image API directly to production

Rejected because: Shadow testing and manual review exposed false positives, so DoorDash selected a more granular vision model and tuned thresholds first.

Patterns

model-routing (moderation-cascade)

SafeChat routes most messages through cheap or internal classifiers and reserves expensive precise LLM review for the small fraction of risky or ambiguous content.

fine-tuning-adaptation (custom-model-optimization)

DoorDash used production labels from the external cascade to train an internal model that replaced the first two external layers.

data-quality-loop (moderation-evaluation)

The image pipeline used shadow testing, manual review, curated datasets, and threshold tuning before production deployment.

canary-rollout (voice-safety)

Voice moderation was first enabled without automatic actions so the team could review flagged conversations and refine thresholds before enforcement.

human-in-the-loop-gate (safety-enforcement)

Severe or repeated violations escalate to human safety agents after automated detection and graduated interventions.

Tradeoffs

How to balance cost, latency, and precision in text moderation.

Chose a cascading model architecture over sending every message to the slowest and most precise LLM.

Rationale: The cascade clears most safe messages cheaply and quickly, while using expensive review only where risk justifies it.

How to expose automated moderation actions to users.

Chose graduated responses that sometimes give users a penalty-free cancellation option over only applying opaque automated enforcement.

Rationale: This combines automated risk detection with user agency and fairness in potentially unsafe delivery interactions.

Gotchas (what bit them)

A broad image safety API created too many false positives.

Root cause: The initial image API returned coarse unsafe/safe categories that could not distinguish diverse image-safety cases well enough.

Resolution: DoorDash switched to a more granular model and tuned thresholds against curated positive and false-positive examples.

External LLM cascades were too expensive for sustained high-volume moderation.

Root cause: The first architecture relied on several external models even though most traffic was safe.

Resolution: Production labels bootstrapped an internal classifier that handles most messages under lower latency and cost.

Voice moderation could not prevent unsafe content before it was heard.

Root cause: Voice is an asynchronous or in-progress channel where inline blocking is less straightforward than chat/image delivery.

Resolution: SafeChat uses reactive call-dropping, contact prevention, and a phased rollout with manual review before automated enforcement.

Evaluation

DoorDash evaluated SafeChat through production labels, shadow testing, manual review of flagged/unflagged samples, curated threshold-tuning datasets, latency distributions, and safety-incident outcomes.

Metrics: messages cleared by each layer · false positive rate · latency · threshold performance · safety incident reduction · model cost

Outcomes

• SafeChat reduced low- to medium-severity safety incidents by 50%.
• The internal model handles 99.8% of messages alone.
• Most message moderation finishes under 300 ms.
• Only flagged messages experience up to about three seconds of latency.
• Only about 0.2% of text content reaches the most intensive analysis in the earlier cascade.

SafeChat processes millions of daily messages, hundreds of thousands of images, and voice calls exchanged between Dashers and consumers.