Strategic Model Management and Multi-Provider Optimization at Scale

notion · Tech · 2026 · source Agentic: yes · promotional bias: med · summary-only (no fetched source) · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Large-scale LLM procurement, routing, and cost optimization

Notion needed to serve AI features for millions of users while navigating volatile model pricing, provider deprecations, frontier labs that are both suppliers and competitors, and workload differences between routine tasks and frontier-reasoning tasks.

Constraints:

• Frontier model pricing is opaque, volatile, and can change through token usage, rate changes, or deprecations.
• Supplier labs can also build competing first-party products with lower inference COGS.
• The best model changes rapidly as new models and tool requirements appear.
• Architecture, orchestration, compaction, caching, and context management can swing costs beyond model selection alone.

Architecture

Notion built a multi-provider LLM operating model centered on optionality. Its Auto model feature evaluates major models on cost-per-capability-per-second and switches approximately 75% of AI traffic every two to three weeks as model performance and costs change. In parallel, Notion optimizes orchestration, compaction, caching, and context management, and invests in open-weight models as credible alternatives for moderate workloads.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  notion_ai_traffic["Notion AI traffic"] --> task_segmentation_layer["Task segmentation layer"]
  task_segmentation_layer --> auto_model_router["Auto model router"]
  auto_model_router --> provider_evaluation_scorecards["Provider evaluation scorecards"]
  provider_evaluation_scorecards --> frontier_model_providers["Frontier model providers"]
  auto_model_router --> frontier_model_providers
  task_segmentation_layer --> open_weight_model_alternatives["Open-weight model alternatives"]
  open_weight_model_alternatives --> inference_providers["Inference providers"]
  architecture_cost_layer["Architecture cost layer"] --> auto_model_router
  architecture_cost_layer --> task_segmentation_layer

Components

Data flow

AI traffic enters Notion's model-management layer. Workloads are segmented by task difficulty and value, evaluated against model scorecards, routed through the Auto feature to the best provider/model for the task, and periodically switched as market and eval results change. Architecture-level optimizations such as caching and context management reduce cost independent of provider choice.

Why this architecture

Notion maintains multi-provider optionality because lock-in weakens negotiation leverage and can strand customers on yesterday's best model. Auto routing, eval scorecards, open-weight alternatives, and architecture-level optimizations let Notion match capability to task while preserving customer trust and cost control.

Models used: frontier models · Claude Code · Haiku · Nano · open-weight LLMs

Scale

• The Auto model feature handles approximately 75% of Notion AI traffic.
  “The operational centerpiece of Notion's LLMOps approach is the "Auto" model feature, which handles approximately 75% of their AI traffic.”summary, stated
• Notion switches models every two to three weeks as evaluations and model releases change.
  “The evaluation and switching happens rapidly: every two to three weeks, as new models are released”summary, stated
• Architecture decisions can swing costs by 3x.
  “These architectural decisions can swing costs by 3×, making architecture as important as model choice for cost management.”summary, stated
• The summary describes Notion's traffic as massive and diverse across horizontal knowledge work.
  “The company's diverse, massive AI traffic across horizontal knowledge work provides unique value to labs”summary, stated

Alternatives considered (roads not taken)

Commit to one frontier lab for upfront discounts

Rejected because: Notion believes optionality creates better long-term price, trust, and model-fit outcomes than single-provider lock-in.

Use frontier models for all moderate tasks

Rejected because: Moderate tasks often do not improve with more powerful models, so cost rather than frontier capability moves the needle.

Optimize only model selection

Rejected because: Model choice accounts for only part of controllable cost; architecture work can swing costs by 3x.

Patterns

model-routing (serving)

Notion's Auto feature evaluates major models and routes traffic to the right model for each task, changing choices as the market changes.

cost-aware-routing (cost optimization)

The architecture explicitly balances capability, cost, latency, provider optionality, open-weight alternatives, caching, compaction, and orchestration.

eval-driven-development (model operations)

Notion uses detailed eval scorecards and provider feedback to decide when and why to switch models.

context-compaction (context management)

Compaction is named as one of the infrastructure levers that reduces prompt/context cost beyond model choice.

semantic-caching (cost optimization)

Caching is part of Notion's architecture-layer cost control rather than relying only on provider negotiation.

Tradeoffs

Preserve multi-provider optionality.

Chose Work with all model providers and keep walk-away leverage over Single-provider discounts and annual lock-in.

Rationale: Notion believes optionality produces better long-term trust, growth, and customer model fit than upfront discounts.

Segment tasks by capability need.

Chose Open-weight and cost-optimized models for moderate tasks, frontier models for high-value reasoning workflows over One model tier for all traffic.

Rationale: Moderate tasks saturate below frontier capability, while autonomous agents, data analysis, research, and remediation genuinely need stronger models.

Build product value around models.

Chose UI, orchestration, architecture, integrations, and product experience over Competing on raw model training or per-token rate alone.

Rationale: Notion's strategy is to make the token a commodity input and win on the surrounding product experience.

Gotchas (what bit them)

Provider lock-in can trap applied AI products on bad economics.

Root cause: Frontier labs control pricing, deprecations, and competing first-party products while selling inference to applied AI companies.

Resolution: Maintain multi-provider optionality and credible open-weight alternatives.

Rate cards hide true model economics.

Root cause: Per-token pricing can stay constant while reasoning-token usage, deprecations, or task fit make effective cost rise.

Resolution: Evaluate cost per capability per second and treat rate cards as negotiation starting points.

Architecture can dominate model-choice savings.

Root cause: Redundant calls, poor orchestration, inefficient context, and missing caching can inflate costs regardless of provider selection.

Resolution: Invest in boring infrastructure such as orchestration, compaction, caching, and context management.

Evaluation

Notion evaluates models continuously through cost-per-capability-per-second scorecards across diverse real-world knowledge-work tasks, then uses results to switch models and provide feedback to providers.

Metrics: Approximately 75% of AI traffic handled by Auto. · Model switching every two to three weeks. · Cost-per-capability-per-second evaluation. · Architecture choices can swing controllable costs by 3x.

Outcomes

• Notion can move customers across state-of-the-art models without manual customer intervention.
• The Auto feature handles about 75% of AI traffic.
• Architectural optimization can reduce controllable costs by up to 3x.

The case describes millions of users, approximately 75% AI traffic on Auto, switching every two to three weeks, and architecture-level cost swings up to 3x.