Scaling AI Product Development with Rigorous Evaluation and Observability

notion · Tech · year n/a · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Evaluation and observability for large-scale AI product development

Notion AI needed to ship polished AI features for a 100M+ user workspace while switching models quickly, supporting multilingual workspaces, and avoiding regressions across meeting notes, search, and deep research features.

Constraints:

• Notion's product brand requires polish, while AI products often fail in edge cases or demos-only conditions.
• New foundation models need to be evaluated and moved into production quickly.
• The workspace and search index change continuously, making static golden datasets brittle.
• Most Notion AI engineers speak English, while a majority of enterprise users are non-English speakers.
• Human evaluators and data specialists need tooling to inspect traces, datasets, and judge prompts at scale.

Architecture

Notion uses Brain Trust as an evaluation and observability layer around AI product development. Product teams, data specialists, and engineers build handcrafted datasets, product-specific scoring functions, LLM-as-judge prompts, and observability traces. Modular prompts and model assignments let the team test new foundation models quickly, route cheaper models to suitable high-frequency tasks, and keep multilingual/search behavior measurable.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Notion AI product surfaces"] --> B["Production observability traces"]
  B --> C["Brain Trust evaluation platform"]
  C --> D["Data specialist workflow"]
  D --> E["LLM judge prompts"]
  E --> C
  C --> F["Prompt and model router"]
  F --> G["Foundation and fine-tuned models"]
  F --> A

Components

Data flow

Production usage and dogfooding traces flow into Brain Trust, where data specialists create small targeted datasets and scoring functions. Product-specific LLM judges score outputs, CI and eval runs compare candidate prompts or models, and observability traces reveal production behavior. Model-selection decisions then map prompts to the model that best fits cost, latency, and reasoning needs.

Why this architecture

Notion treats evaluation and observability as the core IP of AI product development because prompt strings alone do not guarantee consistent quality at product scale.

Models used: foundation model providers · Notion fine-tuned models · Nano · LLM-as-a-judge models

Scale

• Notion has over 100 million users.
  “we have a 100 million users. Um over 100 million users now.”transcript, stated
• Notion can expose new model releases to production users in less than a day.
  “Anytime a new model is released within um, usually less than a day, we're able to give that to all of our users in production.”transcript, stated
• The team reports spending about 10% of time on prompting and 90% on evals, iteration, usage, and observability.
  “we spend maybe 10% of our time prompting and 90% of our time looking at evals and iterating on our evals”transcript, stated
• Sixty percent of Notion enterprise users are non-English speakers.
  “60% of notion enterprise users aren't speaking English. I will tell you a 100% of notion AI engineers are speaking English.”transcript, stated

Alternatives considered (roads not taken)

Generic one-prompt LLM-as-judge evaluation

Rejected because: Notion finds per-element, product-specific judge prompts more insightful for expected format, language, and search behavior.

Static golden outputs for changing search results

Rejected because: The underlying index changes, so criteria-based eval prompts age better than fixed expected answers.

Use advanced models for every prompt by default

Rejected because: Notion evaluates cheaper, faster models for high-frequency lower-reasoning cases before switching prompts.

Patterns

eval-driven-development (evaluation)

Notion centers AI product development around eval datasets, scoring functions, CI-style eval runs, and model regression checks.

observability-tracing (operations)

Usage and traces in Brain Trust are used to understand production AI behavior and guide iteration.

llm-as-judge (evaluation)

Notion uses multiple LLM judges and targeted judge prompts instead of a single generic evaluator.

model-routing (model-ops)

Prompts are modularly assigned to different models after eval runs identify which tasks can move to cheaper or faster models.

data-quality-loop (data)

Data specialists use logs and prototypes to create handcrafted datasets and improve eval structure.

Tradeoffs

Evaluation strategy

Chose laborious product-specific judge prompts over one generic LLM-as-judge prompt.

Rationale: Specific prompts better capture expected language, formatting, citation, and search behavior.

Model rollout strategy

Chose eval-gated rapid model switching over manual slow migration to new models.

Rationale: The team can detect regressions and move suitable prompts to new models quickly.

Gotchas (what bit them)

AI demos can work once but fail for real users.

Root cause: Prompt success without observability and evals does not show consistency across user scenarios.

Resolution: Spend most development effort on evals, usage analysis, and observability.

Human evaluator workflows can collapse under raw trace volume.

Root cause: Data specialists were trying to inspect prompt dumps and parse data in spreadsheets.

Resolution: Move data specialists into Brain Trust workflows with structured datasets and scoring functions.

Multilingual quality is hard when the engineering team is English-speaking.

Root cause: A large share of enterprise users use non-English workspaces.

Resolution: Use evaluation metrics that explicitly understand the multilingual experience.

Evaluation

Notion uses Brain Trust datasets, product-specific LLM judges, scoring functions, CI-style eval runs, trace observability, and human data specialists.

Metrics: model regression scores · search result correctness · multilingual quality · tool-call appropriateness · production usage and traces

Outcomes

• Notion can integrate new model releases in less than a day.
• Brain Trust became central to model selection, evals, and observability.
• Data specialists moved from spreadsheet-like trace work to structured eval datasets.
• Multilingual quality became measurable despite an English-speaking engineering team.

The system supports a connected workspace with over 100 million users and AI surfaces including meeting notes, enterprise search, and deep research.