A Practical Blueprint for Evaluating Conversational AI at Scale

dropbox · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Conversational AI evaluation

Dropbox needed repeatable evaluation for Dropbox Dash because pipeline changes could regress intent classification, retrieval, ranking, prompts, inference, or safety.

Constraints:

Architecture

Dropbox Dash is framed as a multi-stage conversational AI pipeline wrapped by a first-class evaluation system. The product path includes intent classification, document retrieval, ranking, prompt construction, LLM inference, and safety filtering; the evaluation path surrounds that pipeline with public and internal datasets, LLM-as-judge rubrics, Braintrust experiment management, pull-request/staging/live gates, dashboards, and low-scoring trace mining. The result is less a single model change than an evaluation operating system for safely changing prompts, retrievers, models, and context selection.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  dropbox-dash-conversational-pipeline[Dropbox Dash conversational pipeline] --> intent-classifier[Intent classifier]
  intent-classifier --> document-retriever[Document retriever]
  document-retriever --> ranker[Ranker]
  ranker --> prompt-constructor[Prompt constructor]
  prompt-constructor --> model-inference[Model inference]
  model-inference --> safety-filter[Safety filter]
  evaluation-datasets[Evaluation datasets] --> llm-judge[LLM judge]
  llm-judge --> braintrust[Braintrust]
  braintrust --> github-action-gates[GitHub Action gates]
  braintrust --> dashboards[Dashboards]
  dropbox-dash-conversational-pipeline --> live-traffic-scorer[Live traffic scorer]
  live-traffic-scorer --> dashboards
  document-retriever --> trace-level-debugging[Trace-level debugging]
  trace-level-debugging --> braintrust
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Dropbox Dash conversational pipeline User-facing conversational AI system whose stages can regress independently. Intent classifier, Document retriever, Ranker, Prompt constructor, Model inference, Safety filter
Intent classifier Classifies the user request before retrieval and answer generation. Dropbox Dash conversational pipeline, Document retriever
Document retriever Finds candidate source material and connector content for the answer. Intent classifier, Ranker, Trace-level debugging
Ranker Orders retrieved material before prompt construction. Document retriever, Prompt constructor
Prompt constructor Builds the model prompt from query, retrieved context, and settings. Ranker, Model inference
Model inference Generates the conversational answer. Prompt constructor, Safety filter, LLM judge
Safety filter Final product-path guard before answers are returned. Model inference
Evaluation datasets Public, internal, representative, and synthetic datasets used for regression and sweep coverage. LLM judge, Braintrust
LLM judge Scores answer correctness, citations, formatting, tone, and other rubric dimensions. Evaluation datasets, Braintrust, Model inference
Braintrust Versioned evaluation platform for datasets, scorers, experiments, automation, tracing, and monitoring. Evaluation datasets, LLM judge, Dashboards
GitHub Action gates Runs canonical evaluation queries for pull-request and merge checks. Braintrust, Dashboards
Live traffic scorer Asynchronously scores sampled production traffic without adding user latency. Dropbox Dash conversational pipeline, Dashboards
Dashboards Expose pass/fail rates, drift, latency, cost, and rolling quality metrics to engineering and product. Braintrust, GitHub Action gates, Live traffic scorer
Trace-level debugging Drills into retrieval hits, prompt payloads, generated answers, and judge critiques. Document retriever, Braintrust

Data flow

A Dash query moves through intent classification, retrieval, ranking, prompt construction, inference, and safety filtering. Proposed changes to any of those stages are evaluated against curated public/internal datasets, scored by rubrics and LLM judges, tracked in Braintrust runs, gated in pull requests and staging, then monitored by asynchronous live-traffic scoring once deployed.

“behind that minimalism runs a chain of probabilistic stages, including intent classification, document retrieval, ranking, prompt construction, model inference, and safety filtering.”article, stated
“Every new model version, retriever setting, or prompt change was checked against these dimensions. If performance slipped below the thresholds, the change didn’t move forward.”article, stated

Why this architecture

Dropbox chose layered evaluation because ad-hoc testing could not control regressions in a probabilistic multi-stage pipeline. A versioned platform and automated gates made prompt, retriever, model, and context-selection changes reproducible, comparable, and enforceable before they reached users.

“We designed and built a standardized evaluation process that treated every experiment like production code. Our rule was simple: Handle every change with the same care as shipping new code.”article, stated
“If two people ran the same test with slightly different prompts or model versions, there was no reliable way to track what changed or why.”article, stated

Models used: LLM judges

Scale

Alternatives considered (roads not taken)

Traditional lexical and embedding metrics

Rejected because: They caught extreme drift but missed citation gaps, hallucinated filenames, formatting errors, and factual failures.

“We used these metrics early on for quick checks, and they were useful for catching egregious cases where the model drifted wildly. But they couldn’t enforce deployment-ready correctness.”article, stated

Public benchmark datasets alone

Rejected because: Public datasets lacked the long-tail phrasing and content mix of real Dash users.

“But public datasets alone aren’t enough. To capture the long tail of real-world phrasing, we turned to internal datasets by collecting production logs from Dropbox employees dogfooding Dash.”article, stated

Spreadsheet-managed experiments

Rejected because: Spreadsheets made results scattered, hard to reproduce, and hard to compare across model or prompt versions.

“Spreadsheets were fine for quick demos, but they broke down fast once real experimentation began. Results were scattered across files, hard to reproduce, and nearly impossible to compare side by side.”article, stated

Only offline or staging evaluation

Rejected because: Dropbox needed asynchronous live-traffic scoring to catch silent degradation from real user queries without adding latency.

“we continuously sampled live production traffic and scored it with the same metrics and logic as our offline suites.”article, implied

Patterns

eval-driven-development (evaluation)

Experimental changes are treated like production code and measured against curated datasets and rubrics.

“**We approached our LLM judges as if they were software modules: designed, calibrated, tested, and versioned.”article, stated

llm-as-judge (evaluation)

Judge models assess factual correctness, citation use, formatting, and tone when lexical metrics are insufficient.

“A judge model can check for factual correctness against ground truth or context, assess whether every claim is properly cited, enforce formatting and tone requirements, and scale across dimensions that traditional metrics simply ignore.”article, stated

Tradeoffs

Treat model, retrieval, and prompt changes as gated code changes.

Chose Automated evaluation gates on every material pipeline change over Faster ad-hoc experimentation without hard regression thresholds.

Rationale: Dropbox accepted extra evaluation overhead because conversational AI regressions could appear in any stage of the pipeline.

“Every new model version, retriever setting, or prompt change was checked against these dimensions. If performance slipped below the thresholds, the change didn’t move forward.”article, stated

How to score conversational AI quality.

Chose LLM judges with calibrated rubrics over BLEU, ROUGE, BERTScore, and embedding similarity alone.

Rationale: Dropbox needed factuality, citation, formatting, and tone checks that traditional text metrics did not enforce.

“A judge model can check for factual correctness against ground truth or context, assess whether every claim is properly cited, enforce formatting and tone requirements”article, stated

Where to catch production regressions.

Chose layered merge, stage, and production gates over one large evaluation pass late in release.

Rationale: Layered gates let Dropbox block obvious regressions early and catch scale-only issues in production-like or live settings.

“we used layered gates that gradually tightened requirements and brought the evaluation environment closer to real-world usage.”article, stated

Gotchas (what bit them)

Traditional text metrics did not enforce deployment-ready correctness.

Root cause: BLEU/ROUGE-like metrics missed citation and factual failures.

Resolution: Dropbox added LLM-based rubrics and judge calibration.

“But they couldn’t enforce deployment-ready correctness.”article, stated

Evaluation

The case itself is an evaluation architecture, combining curated datasets, rubrics, judge models, and development workflows.

Metrics: factual correctness · citation correctness · formatting · tone

“We captured these lessons in a playbook that covers the full arc of datasets, metrics, tooling, and workflows.”article, stated

Outcomes

The system is positioned as a reusable playbook for multi-modal conversational AI evaluation.

“They developed a systematic evaluation-first methodology treating every experimental change like production code, requiring rigorous testing before merging.”summary, stated