Scaling Data Infrastructure for AI Features and RAG

notion · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: data infrastructure for AI and search

Notion needed a scalable data lake to support fast-growing block data, update-heavy workloads, analytics, search, and AI/RAG features.

Constraints:

• Data volume grew 10x in three years and doubled every 6 to 12 months.
• The workload was dominated by updates, while Snowflake and many warehouses are optimized for insert-heavy ingestion.
• AI and Search required denormalized block and permission data without overloading live production databases.

Architecture

Notion built an in-house data lake using Debezium CDC connectors on EKS, Kafka, Apache Hudi Deltastreamer, Spark, and S3. Incremental Postgres changes flow through Kafka and Hudi into S3, where Spark jobs transform and denormalize raw data before downstream analytics, Search, AI, vector database, and key-value systems consume cleaned datasets.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Postgres shards"] --> B["Debezium CDC connectors"]
  B --> C["Kafka topics"]
  C --> D["Hudi Deltastreamer"]
  D --> E["S3 data lake"]
  A --> G["RDS snapshot export"]
  G --> E
  E --> F["Spark processing jobs"]
  F --> H["Downstream product stores"]
  E --> H

Components

Data flow

Postgres changes are captured by Debezium connectors, written to Kafka topics, consumed by Hudi Deltastreamer into S3, transformed and denormalized by Spark, then written back to S3 or downstream systems for analytics, reporting, AI, Search, and product features. Rare full snapshots bootstrap new tables, with Kafka replay preserving changes made during the snapshot.

Why this architecture

Notion chose a data-lake architecture because update-heavy block data, permission denormalization, and AI/Search freshness needs outgrew warehouse-centric ingestion. S3 provided cheap scalable storage, Spark provided flexible transformations, and Hudi matched Debezium CDC for update-heavy ingestion.

Scale

• Notion data expanded 10x in three years.
  “In the past three years Notion’s data has expanded 10x due to user and content growth, with a doubling rate of 6-12 months.”article, stated
• Notion grew from more than 20 billion to more than 200 billion block rows.
  “At the start of 2021 we had more than 20 billion block rows in Postgres, and this figure has since grown to more than two hundred billion blocks”article, stated
• Ninety percent of Notion upserts are updates.
  “This causes block data to be predominantly update-heavy—90% of Notion upserts are updates.”article, stated
• The CDC setup handled tens of MB/sec of Postgres row changes as of May 2024.
  “As of May 2024, it smoothly handles tens of MB/sec of Postgres row changes.”article, stated
• Notion preserved a 480-shard partitioning scheme in S3.
  “This partitions the S3 dataset into 480 shards, from shard0001 to shard0480, making it more likely that a batch of incoming updates map to the same set of files”article, stated
• The data lake saved over a million dollars in 2022.
  “Moving several large, crucial Postgres datasets (some of them tens of TB large) to data lake gave us a net savings of over a million dollars for 2022”article, stated

Alternatives considered (roads not taken)

Continue Snowflake-centric ingestion

Rejected because: Snowflake ingestion became slower and more costly for Notion's update-heavy workload.

Periodic full snapshots as the normal path

Rejected because: Full snapshots took more than 10 hours and cost twice as much, so Notion used them only for rare bootstraps.

Iceberg or Delta Lake for ingestion

Rejected because: At the time, they were less suited to the Debezium/update-heavy requirements or lacked an open-source Debezium-ready path.

Patterns

batch-offline-processing (data-infrastructure)

Spark jobs transform, denormalize, and enrich raw S3 data for analytics, reporting, Search, and AI features.

rag-retrieval (retrieval)

The data lake became foundation infrastructure for Search and AI Embedding RAG systems.

Tradeoffs

Data repository

Chose S3 as raw and processed data lake over keeping all heavy ingestion and compute in Snowflake/product stores.

Rationale: S3 fit Notion's AWS stack, storage scale, Spark processing, cost, and downstream product requirements.

Ingestion strategy

Chose incremental CDC plus rare snapshots over periodic full snapshots.

Rationale: Incremental ingestion is fresher and cheaper, while snapshots are useful only for bootstrap and resync scenarios.

Lake table format

Chose Apache Hudi over Apache Iceberg or Delta Lake.

Rationale: Hudi matched the update-heavy workload and provided native Debezium CDC integration.

Gotchas (what bit them)

Warehouses struggled with Notion's update-heavy workload.

Root cause: Most data warehouses are optimized for inserts, while Notion's block data is dominated by updates.

Resolution: Use Debezium CDC, Kafka, and Hudi to apply changed rows into S3.

Permission-aware AI/Search datasets required expensive denormalization.

Root cause: Block permissions are not statically stored and require tree traversal computation.

Resolution: Move raw data into S3 and use Spark transformations for denormalized product datasets.

Full snapshots were too expensive and slow for routine ingestion.

Root cause: Dumping full Postgres snapshots to S3 took more than 10 hours and cost twice as much as incremental ingestion.

Resolution: Use incremental CDC during normal operation and reserve snapshots for bootstrapping new S3 tables.

Evaluation

Notion evaluated the infrastructure through ingestion latency, operational smoothness, cost savings, ability to handle data growth, and successful rollout of AI/Search features.

Metrics: data growth rate · row-change throughput · ingestion delay · resync time · cost savings · AI/Search feature enablement

Outcomes

• End-to-end ingestion time fell from more than a day to minutes for small tables and up to a couple of hours for large ones.
• Several large Postgres datasets moved to the lake, saving over a million dollars in 2022 with proportionally higher savings afterward.
• The data lake enabled Notion AI features in 2023 and 2024, including Search and AI Embedding RAG infrastructure.

The system handles hundreds of billions of block rows, 480 shards, and tens of MB/sec of row changes.