Building Custom Agents at Scale: Notion's Multi-Year Journey to Production-Ready Agentic Workflows

notion · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise custom agents and workspace automation

Notion needed to make custom agents reliable enough to run background workflows across enterprise workspaces without exposing users, administrators, or models to the wrong abstractions. The system had to support many Notion tools, workspace permissions, user-created agents, agent composition, usage-based pricing, evaluation at release quality, and retrieval behavior optimized for agents rather than only humans.

Constraints:

• Early models lacked robust tool calling, had short context windows, and were not reliable enough for background work.
• Agent tools had to respect enterprise sharing, Slack-channel context, document permissions, and administrator expectations.
• The tool surface grew beyond 100 tools, making prompt size, tool discovery, and tool ownership bottlenecks.
• Notion's underlying product abstractions were not always model-friendly, so lossless internal representations could hurt model behavior.
• Costs had to map to tokens, model tier, web search, serving tier, and future sandbox usage rather than only agent runs.

Architecture

Notion built Custom Agents as a production agent harness over Notion's existing system of pages, databases, tools, integrations, and permissions. After several rebuilds, the harness moved from JavaScript-coding agents and custom XML toward model-friendly Notion-flavored markdown, SQL-like database queries, declarative tool definitions, and progressive tool disclosure through tool search. Product teams own the tools their surfaces expose to agents, while the platform team owns eval frameworks, model behavior infrastructure, and the shared agent harness. Agents can set themselves up, write or revise their own instructions, use setup tools, operate over pages and databases as memory, invoke other agents, and coordinate through databases or webhooks. The production system is guarded by explicit permissions, security involvement from the beginning, usage-based credits, model auto-selection, eval tiers, and retrieval/ranking investments for agent-generated search traffic.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  custom-agent-chat[Custom agent chat] --> setup-tools[Setup tools]
  custom-agent-chat --> tool-discovery[Tool discovery]
  custom-agent-chat --> permission-guardrails[Permission guardrails]
  setup-tools --> agent-instructions[Agent instructions]
  setup-tools --> notion-databases[Notion databases]
  agent-instructions --> tool-discovery
  permission-guardrails --> custom-agent-chat
  tool-discovery --> tool-registry[Tool registry]
  tool-registry --> product-owned-tools[Product-owned tools]
  tool-registry --> mcp-integrations[MCP integrations]
  tool-registry --> custom-integrations[Custom integrations]
  product-owned-tools --> notion-flavored-markdown[Notion-flavored markdown]
  product-owned-tools --> sql-like-database-interface[SQL-like database interface]
  notion-flavored-markdown --> pages[Pages]
  sql-like-database-interface --> notion-databases
  notion-databases --> manager-agent[Manager agent]
  notion-databases --> specialized-agents[Specialized agents]
  manager-agent --> specialized-agents
  manager-agent --> agent-issue-database[Agent issue database]
  specialized-agents --> agent-issue-database
  specialized-agents --> webhooks[Webhooks]
  eval-framework[Eval framework] --> regression-evals[Regression evals]
  eval-framework --> launch-evals[Launch evals]
  eval-framework --> notion-last-exam[Notion Last Exam]
  eval-framework --> model-behavior-engineers[Model behavior engineers]
  credits-and-model-auto[Credits and model auto] --> model-portfolio[Model portfolio]
  agentic-retrieval-stack[Agentic retrieval stack] --> pages
  agentic-retrieval-stack --> notion-databases

Components

Data flow

A user configures or invokes a custom agent through chat. The setup flow gives the agent development guidance and setup tools so it can draft its own instructions, configure databases, and preview settings, while permission changes require explicit human/admin confirmation. During runtime, the agent discovers relevant tools through progressive disclosure rather than receiving all 100+ definitions, then operates through Notion-flavored markdown, SQL-like database queries, native integrations, MCP integrations where appropriate, and custom-built Slack/Gmail tools where deeper triggers or tool tuning are needed. Agents persist memory by editing pages or databases, coordinate by writing tasks to shared databases or invoking other agents through webhooks, and can be supervised by manager agents. Evaluation flows run in CI, nightly, launch report cards, and frontier/headroom suites; model behavior engineers and eval agents inspect failures and guide product/model changes. Pricing and routing use credits, model tiers, serving tiers, web-search costs, and auto-selection to keep high-cost workflows from overwhelming margins.

Why this architecture

Notion's architecture follows two principles from the source: give models model-friendly interfaces, and keep enterprise control explicit. The team abandoned Notion-internal XML and JSON abstractions in favor of markdown and SQL-like query surfaces because models already understand them. It moved from few-shot prompt strings to tool definitions and progressive disclosure because the tool surface was too large for one prompt and needed distributed ownership by product teams. It kept memory as pages/databases and agent coordination as databases/webhooks because those are Notion's native primitives. Finally, it brought security, evals, pricing, and model behavior work into the core loop because custom agents run in the background and must be trusted by enterprise users.

Models used: GPT-4 · Claude Sonnet 3.5/3.6 · OpenAI GPT-5.2 · MiniMax · Open-source models · Opus · Haiku

Scale

• Custom Agents was roughly the fourth or fifth rebuild of the concept.
  “It was definitely super exciting for me because it’s probably the fourth or fifth time that we rebuilt that.”article, stated
• Notion began the agent effort around GPT-4 access in late 2022.
  “right when we got access to like GPT four in late 20 22, 1 of the first ideas we had”article, stated
• Core AI capabilities and infrastructure included about 50 people.
  “Core AI capabilities and infrastructure. That’s about 50 people.”article, stated
• Packaging teams added another 30 to 40 people.
  “how it shows up in the corner chat versus custom agents versus meeting notes, that’s another 30, 40 people.”article, stated
• The latest agent version had over 100 tools.
  “there’s now like over a hundred tools. Just for all, all the crazy notion stuff.”article, stated
• Launch-quality evals target roughly 80-90% pass rates.
  “we need to, on these categories, you know, be it 80 or 90% of all of these user journeys to launch”article, stated
• Frontier/headroom evals intentionally sit around 30% pass rate.
  “frontier or headroom evals, where we actively wanna be at 30% pass rate.”article, stated
• One internal go-to-market setup used over 30 custom agents.
  “they had built like over 30 custom agents for, uh, for our go to market team”article, stated
• The manager-agent abstraction reduced over 70 daily notifications to about five.
  “instead of 70 notifications per day, it’s like, like five.”article, stated
• A setup-agent launch decision delayed launch by a few weeks to about a month.
  “A few weeks. Yeah, definitely. Like, like a month or so.”article, stated
• Database autofill with Opus on every cell was described as billions of dollars of cost exposure.
  “if every single autofill action was an agent running on Opus on every single database sell, it would be billions of dollars”article, stated
• Agent-generated search is now the majority of search load on AI-enabled plans.
  “The search load and the search traffic. Majority of it’s coming from agents, not humans.”article, stated

Alternatives considered (roads not taken)

Use JavaScript coding agents instead of explicit tools.

Rejected because: The first version tried to give agents JavaScript APIs and have them write code, but the models were not good enough at writing code then.

Expose lossless Notion XML and internal mutation operations.

Rejected because: The representation mapped well to Notion internals but forced the model to learn an unfamiliar XML format and prompt overhead.

Use Notion's internal database JSON query shape.

Rejected because: The internal JSON format mapped to Notion's system but was less model-friendly than SQLite-style queries.

Keep all tool behavior in few-shot prompt examples owned by a central team.

Rejected because: Few-shot prompt ordering and shared-string editing could not scale across product teams; tool definitions enabled distributed ownership.

Put all 100+ tools directly into the prompt.

Rejected because: Showing every tool made simple interactions token-heavy and caused quality issues when niche tools were overcalled.

Use MCP for every integration.

Rejected because: Notion found MCP useful for some tools, but custom Slack/mail integrations needed built-in triggers and heavily tuned tools.

Fine-tune a model to understand Notion tools.

Rejected because: The tools changed too frequently; retraining a model every time would slow down the harness/tool velocity that actually fixed most failures.

Patterns

tool-use-orchestration (architecture)

Notion treats Custom Agents as tool users over Notion APIs, integrations, setup tools, and agent-invocation tools, with product teams owning their tool definitions.

structured-output-contract (architecture)

The harness evolved toward structured tool definitions, Notion-flavored markdown, and SQL-like database queries that are constrained enough for systems but familiar enough for models.

eval-driven-development (evaluation)

Notion uses regression evals, launch report cards, and frontier/headroom evals, with teams owning evals and a platform team maintaining the framework.

agent-memory (architecture)

Notion did not create a separate memory system; agents use pages and databases as editable memory primitives that humans can also inspect and update.

multi-agent-orchestration (architecture)

Users can create many specialized agents, then add manager agents that invoke, observe, and debug subordinate agents through shared databases or direct calls.

safety-policy-enforcement (safety)

Custom Agents default to no permissions, require explicit grants, prevent self-editing of permissions, and involve security early for enterprise compliance.

model-routing (operations)

Notion uses credits and an auto model-selection concept to map tasks to model tiers rather than default every workflow to the most expensive model.

cost-aware-routing (operations)

Credits abstract over token usage, model type, serving tier, web search, and sandbox costs so high-volume agent features can be priced without bankrupting the product.

retrieval-ranking (retrieval)

Notion is reworking search, ranking, indexing, snippets, and query generation because agent queries now dominate search traffic and need different retrieval optimization than human search.

Tradeoffs

Expose model-friendly abstractions rather than lossless internal abstractions.

Chose Notion-flavored markdown and SQLite-like database queries over Lossless XML blocks and Notion's internal JSON database query format.

Rationale: Lossless internal representations made transformations easy for Notion but added unnecessary model-facing complexity; markdown and SQL are formats models already handle well.

Use progressive tool disclosure.

Chose Tool search and short prompts over Sending every tool definition upfront.

Rationale: The 100+ tool surface created token, quality, and ownership pressure; progressive disclosure lets the agent find relevant tools without overloading every request.

Distribute tool ownership to product teams.

Chose Product-owned tool definitions with shared eval/platform support over A central AI team owning one prompt and every tool behavior.

Rationale: Distributed ownership scaled contribution velocity and matched the reality that each product surface owns its agent-facing API.

Keep memory and coordination in Notion primitives.

Chose Pages, databases, webhooks, and agent invocation over A separate dedicated agent-memory or agent-message subsystem.

Rationale: Using native primitives lets humans inspect/edit memory and lets agents coordinate through the same system of record users already understand.

Let setup and usage happen in the same chat.

Chose An agent-centered chat with settings as a side panel over Separate settings UI and usage UI as the primary workflow.

Rationale: This increased bootstrapping power and made the agent itself responsible for setup, test, and workflow execution, but required launch-delaying product work and permission safeguards.

Use credits and model auto-selection for custom agents.

Chose Credits over raw token accounting, with task-appropriate model selection over Flat agent-run pricing or always using the most capable model.

Rationale: Costs include tokens, model tier, serving tier, web search, and future sandboxing; many knowledge-work tasks do not need frontier models.

Use MCP selectively.

Chose MCP for some long-tail or narrow integrations and custom-built integrations for deeply tuned workflows over A single integration strategy for every tool.

Rationale: MCP is useful for some tools and permission shapes, while custom integrations are better when triggers, tool quality, and deterministic operations need deeper product ownership.

Gotchas (what bit them)

Building agents before model capability is ready burns effort unless the team can tell whether it is swimming upstream.

Root cause: Early models were too weak for multi-turn agents, context was too short, and tool standards did not exist yet.

Resolution: Notion treated the work as a portfolio, repeatedly rebuilt, and waited for model capability while learning which parts were engineering versus model limitations.

Internal product representations can be too complex or unnatural for models.

Root cause: Notion first catered to its own block/data model with lossless XML and internal query formats instead of what models had seen in training.

Resolution: The team switched to Notion-flavored markdown and SQLite-like database queries to reduce model-facing complexity.

A central few-shot prompt becomes an organizational bottleneck as tool count grows.

Root cause: Few-shot prompting meant one prompt string, ordering effects, and only a small group allowed to edit safely.

Resolution: Notion moved to tool definitions with eval-backed distributed tool ownership.

Large tool surfaces can both slow the system and degrade model choices.

Root cause: Putting many tools into the prompt made simple interactions expensive and encouraged inappropriate niche-tool calls.

Resolution: The latest harness uses progressive disclosure and tool search while fighting to keep the prompt short.

Self-fixing agents create a permission escalation problem.

Root cause: An agent that can edit its own permissions could undermine the explicit grants that make background delegation trustworthy.

Resolution: Notion lets agents help fix setup in an admin/synchronous mode, but does not let them edit their own permissions silently.

Usage-unlimited background agents can create unacceptable margin risk.

Root cause: Some high-volume automations could invoke expensive frontier models at database scale, and asynchronous users may not care about speed-based incentives.

Resolution: Notion introduced credits, usage-based pricing, UI nudges, and auto model selection to align cost with value and capability.

Human search assumptions do not carry over to agentic retrieval.

Root cause: Agent queries differ from human queries, and ranking objectives shift when the consumer is an agent needing top-K context and snippets rather than click-through position.

Resolution: Notion is reinvesting in retrieval, ranking, indexing, snippets, and query generation as one agentic-find journey.

Evaluation

Notion treats evals as both product gates and an agent harness. Regression evals live in CI or nightly runs, launch-quality report cards need roughly 80-90% pass rates across user journeys, and frontier/headroom evals such as Notion Last Exam intentionally sit near 30% to avoid saturation. Dedicated model behavior engineers, data scientists, and eval engineers inspect failures, write judges, and use coding agents to run evals, debug failures, and implement fixes.

Metrics: Regression eval pass rate within stochastic margins · Launch report-card pass rate by user journey · Frontier/headroom eval pass rate near 30% · Major nightly/CI failures routed to teams · P99 most token-exhaustive custom-agent transcript reviews · Latency-regression acceptance when appropriate · Enterprise-work-focused model snapshot feedback · Tool ownership and eval ownership by product team

Outcomes

• Notion shipped Custom Agents after multiple rebuilds and an alpha period
• the launch was described as the most successful launch for free trials and conversions
• the harness now supports over 100 tools through progressive disclosure
• bug triage custom agents changed internal operating processes by routing Slack reports into task databases
• manager agents reduced one internal user's agent-blocker notifications from over 70 per day to about five
• model behavior engineering became a distinct career path and eval function
• agent-generated search now dominates search traffic on AI-enabled plans
• meeting notes became a high-signal data-capture primitive for agents and search

The case spans late 2022 through the Custom Agents launch, four or five rebuilds, a 50-person core AI capabilities/infrastructure group, 30-40 packaging-team members, over 100 tools, launch evals targeting 80-90%, frontier evals around 30%, 30+ internal go-to-market agents, a manager-agent reduction from 70+ notifications to roughly five per day, a launch delay of weeks to about a month, potential billions of dollars in Opus-backed database-cell autofill cost, and majority search traffic from agents on AI-enabled plans.