Building a Scalable AI Feature Evaluation System

notion · Tech · 2023 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Evaluation infrastructure for production AI features

Notion needed evaluation practices that could keep 70 engineers aligned while AI features moved from simple prompt-and-judge flows to agents with many possible paths, large traces, multilingual edge cases, and rapid frontier-model adoption.

Constraints:

• Agents navigate many possible paths and adjust based on their own results
• Vibe checks do not work at 70-engineer scale
• Regression evals alone may not reveal how a new frontier model behaves differently
• Needle-in-a-haystack customer issues can be hidden in massive traces
• Prompts grew from thousands to hundreds of thousands of tokens

Architecture

Notion uses Braintrust as a core evaluation and observability layer for Notion AI. Feedback and traces feed evaluation datasets, LLM-as-judge checks, custom-code evaluation screens, and searchable trace infrastructure. The team runs regression and frontier evals, builds targeted datasets for issues such as multilingual adherence, uses Brainstore to search massive traces for tool calls and error patterns, and treats evaluation output as the basis for most AI team work.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  notion-ai[Notion AI] --> braintrust[Braintrust]
  notion-ai[Notion AI] --> frontier-models[Frontier Models]
  braintrust[Braintrust] --> evaluation-datasets[Evaluation Datasets]
  evaluation-datasets[Evaluation Datasets] --> llm-judge[LLM Judge]
  braintrust[Braintrust] --> custom-evaluation-code[Custom Evaluation Code]
  custom-evaluation-code[Custom Evaluation Code] --> data-specialist[Data Specialist]
  braintrust[Braintrust] --> brainstore[Brainstore]
  brainstore[Brainstore] --> llm-traces[LLM Traces]
  llm-traces[LLM Traces] --> braintrust[Braintrust]
  frontier-models[Frontier Models] --> braintrust[Braintrust]

Components

Data flow

Customer traces and feedback enter Braintrust. Engineers search traces, build targeted datasets for failures, apply custom evaluation code and LLM judges, compare regression and frontier model behavior, and use the resulting feedback to decide which model capabilities to deploy and which product issues to fix.

Why this architecture

Notion relies on evaluation infrastructure because agentic AI behavior is combinatorial, customer issues hide inside large traces, and frontier models change faster than fixed evaluation methods. Braintrust gave the team custom evaluation code and trace search without forcing a rigid data representation.

Models used: GPT-2 · frontier reasoning models · LLM judges

Scale

• The evaluation framework keeps 70 engineers aligned.
  “a comprehensive framework that keeps 70 engineers aligned while deploying frontier models within hours of release.”article, stated
• Notion started experimenting with LLMs after GPT-2 launched in 2019.
  “Ivan Zhao and Simon Last were early in recognizing the potential of generative AI and started experimenting with large language models (LLMs) soon after GPT-2 launched in 2019. Since then, they've built Notion AI around a few core capabilities: finding”article, stated
• Notion wants access to the latest models within hours of release.
  “One of Notion's AI commitments is giving customers access to the latest models as quickly as possible, ideally within hours of release. This requires both rigorous evals, and a team that can move rapidly from evaluation to production.”article, stated
• 80% of AI team work is based on Braintrust feedback and traces.
  “Now, 80% of what the AI team does is based on evaluating from feedback and traces in Braintrust; tinkering, measuring, and understanding if they are moving in the right direction.”article, stated
• Notion prompts grew from thousands to hundreds of thousands of tokens.
  “As Notion's AI prompts grew from thousands of tokens to hundreds of thousands, the team needed to search not just customer interactions but specific tool calls, error codes, and patterns within massive traces. Standard search was too slow.”article, stated

Alternatives considered (roads not taken)

Use vibe checks at team scale

Rejected because: At 70 engineers and agentic path complexity, subjective checks no longer worked.

Rely on regression evals alone

Rejected because: A frontier model can pass regression evals while behaving differently in ways that matter for deployment decisions.

Use standard search over customer interactions

Rejected because: Notion needed performant search over tool calls, error codes, and patterns within huge traces.

Patterns

eval-driven-development (evaluation)

Notion bases most AI team work on feedback and traces in Braintrust rather than ad hoc checks.

llm-as-judge (evaluation)

Notion uses LLM judges to prevent regressions in targeted capabilities like language adherence.

observability-tracing (observability)

Searchable traces expose customer interactions, tool calls, error codes, and patterns for debugging.

canary-rollout (release)

Frontier evals identify where new models differ so Notion can deploy them for specific use cases quickly.

Tradeoffs

Build both regression and frontier evals

Chose Regression plus frontier evaluation over Regression evals alone.

Rationale: Regression evals catch breakage, while frontier evals reveal model-specific improvements for deployment choices.

Customize evaluation infrastructure

Chose Custom code and Notion-specific data representation on Braintrust over Off-the-shelf evaluation workflows.

Rationale: Notion's horizontal product data required custom UI and labeling flows while still using Braintrust primitives.

Rebuild systems around new model capabilities

Chose Six-month rebuild mindset over Keeping old architecture despite frontier model shifts.

Rationale: New reasoning capabilities required different prompting and system structure.

Gotchas (what bit them)

Agentic evaluation has combinatorial path complexity.

Root cause: Agents adjust based on their own results and can navigate many evaluation paths.

Resolution: Notion moved beyond simple prompt-and-judge setups into trace-backed, dataset-driven evaluation.

Some high-priority quality problems are rare and hard to find.

Root cause: Issues like multilingual language adherence affect subsets of customers and may not show in broad metrics.

Resolution: Braintrust search and evaluation mechanisms let Notion build targeted datasets from failure cases.

Existing databases broke down for large LLM trace search.

Root cause: Prompts expanded to hundreds of thousands of tokens and developers needed specific trace/tool/error searches.

Resolution: Notion adopted Brainstore's LLM-trace indexing infrastructure.

Evaluation

Notion uses Braintrust feedback and traces, regression evals, frontier evals, targeted failure datasets, LLM judges, custom evaluation code, and Brainstore trace search to guide AI development.

Metrics: regression eval pass rate · frontier eval differences · language adherence failures · trace search over tool calls and error codes · 80% AI team work from feedback and traces

Outcomes

• Notion keeps 70 engineers aligned around AI evaluation
• The team can deploy frontier models within hours of release
• Targeted multilingual evals produced one of the top APAC quality improvements in the past year
• Brainstore improved trace search performance for large-context traces

The source emphasizes 70 engineers, prompt traces growing to hundreds of thousands of tokens, and model deployment within hours of release.