One-Click Simulation and Evaluation Platform for Support Chatbots

doordash · E-commerce · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Support chatbot simulation and evaluation

DoorDash needed to shorten support-chatbot validation cycles and catch long-tail failures before exposing LLM changes to customers.

Constraints:

• Previous validation required deploying to 1% of live traffic.
• Manual transcript review took hours to weeks.
• Long-tail support edge cases were difficult to catch.
• Tool calls needed to behave like production while remaining safe in simulation.

Architecture

DoorDash built a one-click white-box simulation and evaluation platform that generates multi-turn production-grounded conversations, routes tool calls through configurable mocks, and grades outcomes with feature-specific LLM-as-a-judge rubrics.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  production-data-grounding["Production Data Grounding"]
  conversation-simulator["Conversation Simulator"]
  support-chatbot["Support Chatbot"]
  configurable-tool-mocks["Configurable Tool Mocks"]
  llm-judge["LLM Judge"]
  production-data-grounding --> conversation-simulator
  conversation-simulator --> configurable-tool-mocks
  conversation-simulator --> support-chatbot
  support-chatbot --> configurable-tool-mocks
  support-chatbot --> llm-judge
  configurable-tool-mocks --> support-chatbot

Components

Data flow

Production data grounds simulated support scenarios, the simulator generates realistic multi-turn conversations, chatbot tool calls are routed through configurable mocks, and LLM-as-a-judge rubrics evaluate feature-specific behavior before live rollout.

Why this architecture

White-box simulation with tool mocks gave teams production-like behavior without risking live users, while rubric-based judging made evaluation feature-specific.

Models used: LLM-powered support chatbot · LLM-as-a-judge

Scale

• Prior validation used 1% of live traffic.
  “deploying changes to 1% of live traffic”summary, stated
• Validation time dropped from seven hours to five minutes.
  “reduced validation time from seven hours to five minutes”summary, stated
• Simulated escalation rates matched production closely: 46% versus 44%.
  “production-like behavior (46% vs 44% escalation rates)”summary, stated
• Simulation hallucinations dropped by 90%.
  “reduced hallucinations in simulations by 90%”summary, stated

Alternatives considered (roads not taken)

Validate through live 1% traffic experiments

Rejected because: It exposed customers to changes and required slow manual transcript review.

Black-box transcript review only

Rejected because: The system needed configurable tool mocks and feature-specific rubrics to catch edge cases before rollout.

Patterns

eval-driven-development (evaluation)

Simulation and LLM-as-a-judge rubrics let teams validate chatbot changes before customer exposure.

llm-as-judge (evaluation)

Feature-specific rubrics are evaluated by an LLM judge.

tool-use-orchestration (testing)

All chatbot tool calls are routed through configurable mocks in simulation.

Tradeoffs

Build white-box simulation

Chose Production-grounded conversations plus tool mocks over Live 1% traffic validation.

Rationale: The platform cut validation time and reduced customer-risk exposure.

Use LLM-as-a-judge rubrics

Chose Feature-specific automated judging over Manual transcript review alone.

Rationale: Automated rubrics made frequent iteration practical.

Gotchas (what bit them)

Live-traffic validation is too slow and risky for LLM chatbot iteration.

Root cause: Teams had to deploy to real users and manually review transcripts.

Resolution: Use production-grounded simulation and rubric evaluation before live exposure.

Simulation can hallucinate unrealistic support behavior.

Root cause: Synthetic conversations can drift from production-like customer/tool dynamics.

Resolution: Ground simulations in production data and route tools through configurable mocks.

Evaluation

DoorDash evaluated validation cycle time, simulated-versus-production escalation rates, and simulation hallucination reduction.

Metrics: Validation time · Escalation-rate parity · Simulation hallucination reduction

Outcomes

• Validation time dropped from seven hours to five minutes.
• Simulated escalation rates aligned with production at 46% versus 44%.
• Simulation hallucinations dropped by 90%.
• Teams could iterate before customer exposure.

The prior release gate used 1% of live traffic; the new platform simulates production-like conversations before release.