Scaling Vector Search Infrastructure for AI-Powered Workspace Search

notion · Tech · 2026 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: workspace vector search infrastructure

Notion needed to scale AI Q&A and workspace search across millions of workspaces while cutting vector database and embedding costs, keeping live workspaces fresh, and reducing user-facing search latency.

Constraints:

• Existing workspaces needed batch onboarding while live edits needed near-real-time indexing.
• The initial vector database coupled storage and compute, making over-provisioning expensive.
• The launch generated a multi-million-workspace waitlist and the original indexes were near capacity within one month.
• Embedding and indexing costs had to fall without breaking permissions, freshness, or query latency.

Architecture

Notion evolved from Spark batch ingestion plus Kafka realtime indexing into a generation-based vector-index strategy, then migrated from dedicated pod clusters to serverless vector infrastructure and turbopuffer, added page-state hashing to avoid needless re-embedding, and moved embeddings generation and serving toward Ray/Anyscale and Ray Serve with open-source embedding models.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Offline Spark indexing path"] --> C["Vector database generations"]
  B["Kafka realtime indexing path"] --> F["Page state hash cache"]
  B --> C
  C --> D["Serverless vector database"]
  D --> E["Turbopuffer search engine"]
  F --> E
  A --> G["Ray and Anyscale embeddings pipeline"]
  G --> H["Ray Serve embedding endpoint"]
  H --> I["Notion AI Q&A and agents"]
  E --> I
  C --> I

Components

Data flow

Existing documents flow through offline Spark jobs that chunk, embed, and bulk-load vectors; live page edits flow through Kafka consumers; generation IDs route workspace reads and writes to the right vector index set; page-state hashes decide whether text embeddings or only metadata patches need updates; Ray/Anyscale pipelines and Ray Serve move embedding generation and query-time embedding serving into GPU-aware managed compute; Notion AI Q&A and agents search the vector store for workspace context.

Why this architecture

Notion repeatedly separated scale bottlenecks: generation IDs avoided re-sharding pauses during onboarding, serverless and turbopuffer reduced vector-search cost, span-level hashes avoided wasteful re-embedding, and Ray/Ray Serve consolidated embedding compute while reducing third-party API dependency and query latency.

Models used: embedding models · open-source embedding models

Scale

• Notion scaled vector search infrastructure by 10x while reducing costs by 90%.
  “This is the story of how we scaled our vector search infrastructure by 10x while simultaneously reducing costs by 90 percent over the past two years.”article, stated
• The initial launch created a waitlist of millions of workspaces.
  “We quickly accumulated a waitlist of millions of workspaces eager to access Q&A”article, stated
• Daily onboarding capacity increased 600x, active workspaces grew 15x, and vector database capacity expanded 8x.
  3 supporting quotes

  “Daily onboarding capacity: 600x increase”article, stated

  “Active workspaces: 15x growth”article, stated

  “Vector database capacity: 8x expansion”article, stated
• The serverless migration cut peak vector database cost by 50%.
  “A 50 percent cost reduction from peak usage, resulting in several millions of dollars saved annually.”article, stated
• The turbopuffer migration reduced search engine spend by 60%, EMR compute by 35%, and improved p50 query latency from 70-100ms to 50-70ms.
  “60 percent cost reduction on search engine spend”article, stated

  “p50 production query latency improved from 70-100ms to 50-70ms”article, stated
• Page-state hashing reduced data volume by 70%.
  “Through both of these changes, we achieved a 70% reduction in data volume”article, stated

Alternatives considered (roads not taken)

Incremental re-sharding every two weeks

Rejected because: Notion instead provisioned new index generations for new onboarding to avoid repeated reshard operations.

Over-provision to final expected volume

Rejected because: The provider charged for uptime, making over-provisioning prohibitively expensive.

Re-embed and reload an entire page on any edit

Rejected because: Span text and metadata hashes let Notion update only changed embeddings or patch metadata without embedding.

Continue Spark preprocessing plus provider embedding APIs

Rejected because: That created double compute, endpoint reliability dependency, and clunky S3 handoff pipelines.

Patterns

rag-retrieval (retrieval)

Notion AI answers natural-language questions by searching embedded workspace and connected-tool content through vector search.

batch-offline-processing (data-infrastructure)

Notion used Spark batch jobs for existing-document onboarding and later used Ray to pipeline CPU-bound chunking with GPU-bound embedding generation.

semantic-caching (cost-control)

The page-state cache stored span text and metadata hashes so unchanged text could reuse existing embeddings and metadata-only edits could patch the vector database.

Tradeoffs

Use generation-based index routing during hypergrowth.

Chose provisioning new index generations for new workspaces over incremental re-sharding or over-provisioning to final volume.

Rationale: It kept onboarding moving while avoiding recurring re-shard operations and expensive over-provisioned uptime.

Migrate from pod clusters to serverless vector infrastructure.

Chose storage/compute decoupling with usage-based charging over dedicated hardware pod clusters.

Rationale: Serverless removed storage capacity constraints and cut peak costs, but further optimization was still needed.

Switch to turbopuffer.

Chose object-storage-based search with independent namespaces over continuing the existing vector database architecture.

Rationale: The migration reduced search spend and simplified sharding/generation routing for a multi-billion-object workload.

Self-host embedding generation and serving on Ray.

Chose Ray/Anyscale and Ray Serve with open-source embedding models over external embedding APIs and Spark-only preprocessing.

Rationale: Self-hosting removed a third-party API hop and unified CPU/GPU pipeline execution.

Gotchas (what bit them)

The original indexes nearly blocked onboarding one month after launch.

Root cause: Demand created a multi-million-workspace waitlist and the initial pod-cluster/index capacity was close to full.

Resolution: Provision new index generations and route new onboarding there instead of stopping for re-shards.

Coupling storage and compute inflated vector database cost.

Root cause: Dedicated pod clusters charged for database uptime and made capacity planning expensive.

Resolution: Migrate embeddings workload to serverless storage/compute separation and later turbopuffer.

Page edits caused unnecessary re-embedding and vector reload work.

Root cause: The old indexing path re-chunked, re-embedded, and re-uploaded all spans whenever a page or property changed.

Resolution: Use text and metadata hashes to distinguish embedding changes from metadata-only patches.

The near-real-time embedding pipeline depended on external endpoint reliability and hand-rolled pipelining.

Root cause: Spark preprocessing and provider embedding APIs split compute and required S3 batch handoffs to avoid rate limits.

Resolution: Move embedding generation toward Ray/Anyscale with unified CPU/GPU pipelining and self-hosted open-source embeddings.

Evaluation

Notion evaluated migrations through cost reductions, latency changes, onboarding capacity, data-volume reduction, and provider evaluations before committing to search-engine and embedding-platform changes.

Metrics: daily onboarding capacity · active workspace growth · vector database capacity · search engine spend · AWS EMR compute cost · p50 production query latency · data volume

Outcomes

• Vector search infrastructure scaled 10x while total cost fell by 90%.
• Q&A waitlist was cleared by April 2024 after 600x onboarding-capacity growth.
• Serverless migration saved several million dollars annually from peak usage.
• Turbopuffer migration cut search engine spend 60% and improved p50 query latency to 50-70ms.
• Page-state hashing cut data volume by 70%.

The article spans millions of workspaces, a multi-billion-object vector workload, and user-facing query-time embedding latency.