Scaling Vector Search Infrastructure for AI-Powered Workspace Search

notion · Tech · 2026 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Tech

Notion needed vector search infrastructure for AI Q&A to onboard millions of workspaces while controlling rapidly growing vector database and embedding costs.

Constraints:

• millions of workspaces were waiting for Q&A
• initial vector indexes approached capacity soon after launch
• vector database costs were running at millions per year
• query-time retrieval needed low latency

Architecture

Notion's AI Q&A stack uses vector search as the retrieval layer for workspace and connected-tool content. The architecture evolved from dedicated pod-based vector databases with workspace-sharded indexes into a layered indexing and serving system: offline Spark batch jobs and online Kafka consumers generated embeddings and kept workspaces fresh, generation IDs routed reads/writes across new index sets during hypergrowth, serverless and then turbopuffer separated storage economics from compute, Page State hashes avoided redundant re-embedding, and Ray/Anyscale moved embedding generation plus query-time embedding serving onto a self-hosted ML compute layer.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  connected-tool-content["connected-tool-content"] --> offline-indexing-pipeline["offline-indexing-pipeline"]
  connected-tool-content --> online-indexing-consumers["online-indexing-consumers"]
  offline-indexing-pipeline --> embedding-models["embedding-models"]
  online-indexing-consumers --> page-state-cache["page-state-cache"]
  page-state-cache --> offline-indexing-pipeline
  page-state-cache --> online-indexing-consumers
  embedding-models --> vector-search-backend["vector-search-backend"]
  offline-indexing-pipeline --> vector-search-backend
  online-indexing-consumers --> vector-search-backend
  generation-router["generation-router"] --> vector-search-backend
  turbopuffer["turbopuffer"] --> vector-search-backend
  ray-indexing-platform["ray-indexing-platform"] --> embedding-models
  notion-ai-qa["notion-ai-qa"] --> query-embedding-service["query-embedding-service"]
  query-embedding-service --> embedding-models
  query-embedding-service --> vector-search-backend
  vector-search-backend --> notion-ai-qa

Components

Data flow

Notion chunks workspace documents, generates semantic embeddings, and writes vectors plus metadata into a vector search backend. Offline Spark jobs bulk-index existing content while Kafka consumers process live page edits with sub-minute freshness. During growth, workspace generation IDs route reads and writes to the right index set. Later, Page State compares text and metadata hashes from DynamoDB to decide whether to re-embed spans, patch metadata, or skip work. At query time, Notion AI embeds the user query, searches the vector database for semantically relevant workspace/connector content, and uses those retrieved results as context for Q&A and agents.

Why this architecture

Notion chose a retrieval-first vector architecture because AI Q&A depends on semantic search over each workspace and connected tools. The system kept separate offline and online indexing paths so it could bulk-onboard large workspaces while maintaining live freshness. Generation routing let the team keep onboarding while avoiding repeated reshards. Serverless/turbopuffer reduced fixed infrastructure cost, Page State cut redundant write volume, and Ray gave Notion more control over embedding model evolution, reliability, and query-time latency.

Models used: embedding model · open-source embedding models

Scale

• The vector search infrastructure scaled 10x while reducing costs by 90% over two years.
  “we scaled our vector search infrastructure by 10x while simultaneously reducing costs by 90 percent over the past two years.”article, stated
• Demand included a waitlist of millions of workspaces.
  “We quickly accumulated a waitlist of millions of workspaces eager to access Q&A”article, stated
• The first scaling push increased daily onboarding capacity 600x, active workspaces 15x, and vector database capacity 8x.
  3 supporting quotes

  “Daily onboarding capacity: 600x increase”article, stated

  “Active workspaces: 15x growth”article, stated

  “Vector database capacity: 8x expansion”article, stated
• Serverless vector infrastructure cut peak costs by 50% and saved several million dollars annually.
  “A 50 percent cost reduction from peak usage, resulting in several millions of dollars saved annually.”article, stated
• The turbopuffer migration covered a multi-billion-object workload.
  “we committed to migrating our entire multi billion object workload to turbopuffer in late 2024.”article, stated
• The turbopuffer migration reduced search-engine spend 60%, EMR compute 35%, and improved p50 query latency.
  3 supporting quotes

  “60 percent cost reduction on search engine spend”article, stated

  “35 percent reduction in AWS EMR compute costs”article, stated

  “p50 production query latency improved from 70-100ms to 50-70ms”article, stated
• Page State reduced indexing data volume by 70%.
  “Through both of these changes, we achieved a 70% reduction in data volume”article, stated
• Ray was expected to reduce embeddings infrastructure cost by more than 90%.
  “we anticipate a 90+ percent reduction in embeddings infrastructure costs.”article, stated
• The real-time indexing path targeted sub-minute update latency.
  “keeping live workspaces up-to-date with sub-minute latency.”article, stated

Alternatives considered (roads not taken)

Incrementally re-shard existing vector indexes every two weeks

Rejected because: Repeatedly cloning, deleting, and rebalancing indexes would slow onboarding during hypergrowth.

Provision vector capacity for the final expected volume

Rejected because: The provider charged for uptime, making large over-provisioned pod capacity too expensive.

Keep the dedicated pod-based vector database architecture

Rejected because: Pods coupled storage and compute and required capacity planning, while serverless and turbopuffer lowered cost and simplified operations.

Re-embed and reload entire pages after any edit

Rejected because: Most edits only changed part of a page or metadata; span hashes let Notion update only the necessary vectors or metadata.

Continue Spark plus third-party embeddings APIs for near-real-time indexing and query-time embeddings

Rejected because: That design created double compute, endpoint reliability dependence, clunky S3 handoffs, and an extra query-time API hop.

Patterns

rag-retrieval (retrieval)

Vector search supplies semantic retrieval for Notion AI Q&A across workspace and connected-tool content.

cost-aware-routing (operations)

Notion moved vector infrastructure from uptime-priced dedicated pods to usage-priced serverless/object-storage-backed systems to control costs.

batch-offline-processing (data-pipeline)

The migration used full re-indexing and increased offline indexing throughput to rebuild the corpus in turbopuffer.

Tradeoffs

Provision new index sets for onboarding rather than immediately reshaping existing shards.

Chose direct new workspace onboarding to newly provisioned index sets over incremental re-sharding or provisioning to final expected volume.

Rationale: The summary says this avoided the two unattractive options of repeated reshards or expensive final-volume provisioning.

Move from fixed pod capacity toward serverless and object-storage-backed vector search.

Chose serverless vector infrastructure and turbopuffer over dedicated hardware pod clusters.

Rationale: Dedicated pods coupled storage and compute and charged for uptime; serverless/turbopuffer reduced cost and removed capacity constraints.

Track page state hashes before re-embedding.

Chose span-level text and metadata hashes in DynamoDB over re-chunking, re-embedding, and re-uploading every span after any page edit.

Rationale: Hash comparison let Notion distinguish text changes from metadata changes and skip expensive embedding work when possible.

Bring embedding generation and serving onto Ray.

Chose Ray/Anyscale and Ray Serve for embeddings over Spark preprocessing plus external embedding endpoints.

Rationale: Ray reduced double compute, improved model flexibility, and removed a third-party query-time hop at the cost of operating a more direct ML platform path.

Gotchas (what bit them)

Dedicated vector indexes created capacity pressure immediately after launch.

Root cause: Storage and compute were coupled in pod clusters, and demand produced a multi-million-workspace waitlist.

Resolution: Notion first provisioned new index sets for new workspaces, then migrated to serverless and turbopuffer architectures.

Evaluation

Notion evaluated search-engine alternatives and committed to turbopuffer after successful evaluation.

Metrics: cost · latency · capacity · write throughput

Outcomes

• The source reports 10x scaling while reducing costs by 90 percent.

The system supports AI Q&A across Notion workspaces and connected tools.